Running the ziti tunneler on a linux host. If the host is restarted, unable to connect over NetFoundry network. The ziti tunneler process is running post-restart and the NetFoundry console shows the endpoint as ‘green’. Restarting the ziti tunneler service restores the connectivity over NetFoundry. Any ideas on why the ziti tunneler is not correctly connecting after the restart?
It would be helpful if you can share the logs after the reboot with debug enabled.
Hi @William_Penrod - Welcome to the community!. We have a linux package that would run on the host as a service. Are you using this - https://support.netfoundry.io/hc/en-us/articles/9349545637773-How-to-Install-run-ziti-edge-tunneller-as-a-Service?
I don’t seem to be able to upload the log file. This snippet reflects the post reboot logs:
Nov 11 20:10:33 LVT-RPI-EDGE-V311 sh[516]: [ 26216.751] INFO lib/ziti-tunnel-cbs/ziti_hosting.c:601 on_hosted_client_connect() hosted_service[PenState TDC04710 EdgeController-API IP], client[] dst_addr[tcp:10.14.0.81:2101]: incoming connection
Nov 11 20:10:34 LVT-RPI-EDGE-V311 sh[516]: [ 26217.819] INFO lib/ziti-tunnel-cbs/ziti_hosting.c:601 on_hosted_client_connect() hosted_service[PenState TDC04710 EdgeController-DVR IP], client[] dst_addr[tcp:10.14.0.81:2102]: incoming connection
Nov 11 20:10:35 LVT-RPI-EDGE-V311 sh[516]: [ 26218.352] INFO lib/ziti-tunnel-cbs/ziti_hosting.c:601 on_hosted_client_connect() hosted_service[PenState TDC04710 CenterCam-HTTP IP], client[] dst_addr[tcp:10.14.0.82:80]: incoming connection
Nov 11 20:10:35 LVT-RPI-EDGE-V311 sh[516]: [ 26218.691] INFO lib/ziti-tunnel-cbs/ziti_hosting.c:601 on_hosted_client_connect() hosted_service[PenState TDC04710 CenterCam-RTSP IP], client[] dst_addr[tcp:10.14.0.82:554]: incoming connection
Nov 11 20:11:36 LVT-RPI-EDGE-V311 sh[516]: [ 26279.359] WARN lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:618 on_ziti_event() ziti_ctx controller connections failed: api session is partially authenticated, waiting for auth query resolution
Nov 11 20:11:36 LVT-RPI-EDGE-V311 sh[516]: [ 26279.359] INFO programs/ziti-edge-tunnel/ziti-edge-tunnel.c:656 on_event() ztx[/opt/netfoundry/ziti/output.json] status is api session is partially authenticated, waiting for auth query resolution
Nov 11 20:11:36 LVT-RPI-EDGE-V311 sh[516]: [ 26279.359] ERROR programs/ziti-edge-tunnel/ziti-edge-tunnel.c:681 on_event() ztx[/opt/netfoundry/ziti/output.json] failed to connect to controller due to api session is partially authenticated, waiting for auth query resolution
Nov 11 20:43:52 LVT-RPI-EDGE-V311 sh[516]: [ 28215.880] ERROR dk-c-src/library/ziti.c:898 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/connection timed out] from ctrl[https://18.204.45.166:443]
Nov 11 20:43:52 LVT-RPI-EDGE-V311 sh[516]: [ 28215.880] WARN lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:618 on_ziti_event() ziti_ctx controller connections failed: Ziti Controller is not available
Nov 11 20:43:52 LVT-RPI-EDGE-V311 sh[516]: [ 28215.880] INFO programs/ziti-edge-tunnel/ziti-edge-tunnel.c:656 on_event() ztx[/opt/netfoundry/ziti/output.json] status is Ziti Controller is not available
Nov 11 20:43:52 LVT-RPI-EDGE-V311 sh[516]: [ 28215.880] ERROR programs/ziti-edge-tunnel/ziti-edge-tunnel.c:681 on_event() ztx[/opt/netfoundry/ziti/output.json] failed to connect to controller due to Ziti Controller is not available
Nov 11 20:44:02 LVT-RPI-EDGE-V311 sh[516]: [ 28225.927] ERROR lib/ziti-tunnel-cbs/ziti_dns.c:211 ziti_dns_lookup() invalid host lookup[0.debian.pool.ntp.org.local.tld#015]
Nov 11 20:44:02 LVT-RPI-EDGE-V311 named[518]: REFUSED unexpected RCODE resolving '0.debian.pool.ntp.org.local.tld\013/AAAA/IN': 100.64.0.2#53
Nov 11 20:44:02 LVT-RPI-EDGE-V311 named[518]: REFUSED unexpected RCODE resolving '0.debian.pool.ntp.org.local.tld\013/A/IN': 100.64.0.2#53
Nov 11 20:44:03 LVT-RPI-EDGE-V311 sh[516]: [ 28226.259] ERROR d-src/src/tcp_src.c:75 failed to connect: -113(host is unreachable)
Nov 11 20:44:03 LVT-RPI-EDGE-V311 sh[516]: [ 28226.259] ERROR dk-c-src/library/ziti.c:898 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/host is unreachable] from ctrl[https://18.204.45.166:443]
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:682 on_ziti_event() ztx[LVT-PRODUSA PenState-StateCollegePA TDC04710 ZLE-01] router LVT-PRODUSA NetFoundry-AWS-USW2 NERL-02@tls://7ce492e9-57ef-4602-8a2f-8abb84788a1e.production.netfoundry.io:443 disconnected
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR dk-c-src/library/connect.c:853 connect_reply_cb() conn[0.1300] failed to connect [-19/no such device]
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR lib/ziti-tunnel-cbs/ziti_hosting.c:347 on_hosted_client_connect_complete() hosted_service[PenState TDC04710 CenterCam-HTTP IP] client[] failed to connect: connection is closed
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR dk-c-src/library/connect.c:853 connect_reply_cb() conn[0.1298] failed to connect [-19/no such device]
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR lib/ziti-tunnel-cbs/ziti_hosting.c:347 on_hosted_client_connect_complete() hosted_service[PenState TDC04710 CenterCam-HTTP IP] client[] failed to connect: connection is closed
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR dk-c-src/library/connect.c:853 connect_reply_cb() conn[0.1295] failed to connect [-19/no such device]
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR lib/ziti-tunnel-cbs/ziti_hosting.c:347 on_hosted_client_connect_complete() hosted_service[PenState TDC04710 CenterCam-HTTP IP] client[] failed to connect: connection is closed
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR dk-c-src/library/connect.c:853 connect_reply_cb() conn[0.1293] failed to connect [-19/no such device]
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] ERROR lib/ziti-tunnel-cbs/ziti_hosting.c:347 on_hosted_client_connect_complete() hosted_service[PenState TDC04710 CenterCam-HTTP IP] client[] failed to connect: connection is closed
Nov 11 20:44:12 LVT-RPI-EDGE-V311 sh[516]: [ 28235.723] INFO dk-c-src/library/channel.c:731 reconnect_channel() ch[2] reconnecting in 5974 ms (attempt = 0)
Nov 11 20:44:12 LVT-RPI-EDGE-V311 ntpd[556]: error resolving pool 0.debian.pool.ntp.org: Temporary failure in name resolution (-3)
Nov 11 20:44:15 LVT-RPI-EDGE-V311 sh[516]: [ 28238.739] ERROR d-src/src/tcp_src.c:75 failed to connect: -113(host is unreachable)
Nov 11 20:44:15 LVT-RPI-EDGE-V311 sh[516]: [ 28238.739] ERROR dk-c-src/library/connect.c:429 connect_get_net_session_cb() conn[0.0] failed to get session for service[PenState TDC04710 EdgeController-DVR IP]: CONTROLLER_UNAVAILABLE(host is unreachable)
Nov 11 20:44:15 LVT-RPI-EDGE-V311 sh[516]: [ 28238.739] ERROR dk-c-src/library/connect.c:429 connect_get_net_session_cb() conn[0.1] failed to get session for service[PenState TDC04710 CenterCam-HTTP IP]: CONTROLLER_UNAVAILABLE(host is unreachable)
Nov 11 20:44:15 LVT-RPI-EDGE-V311 sh[516]: [ 28238.739] ERROR dk-c-src/library/connect.c:429 connect_get_net_session_cb() conn[0.2] failed to get session for service[PenState TDC04710 Modem-HTTP IP]: CONTROLLER_UNAVAILABLE(host is unreachable)
Nov 11 20:44:15 LVT-RPI-EDGE-V311 sh[516]: [ 28238.739] ERROR dk-c-src/library/ziti.c:898 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/host is unreachable] from ctrl[https://18.204.45.166:443]
Nov 11 20:44:15 LVT-RPI-EDGE-V311 sh[516]: [ 28238.739] ERROR d-src/src/tcp_src.c:75 failed to connect: -113(host is unreachable)
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR dk-c-src/library/model_support.c:67 parse_tokens() jsmn_parse() failed: -2
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR dk-c-src/library/ziti.c:212 ziti_init_opts() dk-c-src/library/ziti.c:184 - load_config(options->config, &cfg) => -13 (Configuration is invalid)
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR programs/ziti-edge-tunnel/ziti-edge-tunnel.c:632 load_id_cb() identity[/opt/netfoundry/ziti/ziti-edge-tunnel] failed to load: failed to initialize ziti
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:715 load_ziti_async() attempting to load ziti instance from file[/opt/netfoundry/ziti/zdns.bash]
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:721 load_ziti_async() loading ziti instance from /opt/netfoundry/ziti/zdns.bash
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO ziti_log_set_level set log level: ziti_log_lvl=3 &ziti_log_lvl = 0x575d2c
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR dk-c-src/library/model_support.c:67 parse_tokens() jsmn_parse() failed: -2
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR dk-c-src/library/ziti.c:212 ziti_init_opts() dk-c-src/library/ziti.c:184 - load_config(options->config, &cfg) => -13 (Configuration is invalid)
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR programs/ziti-edge-tunnel/ziti-edge-tunnel.c:632 load_id_cb() identity[/opt/netfoundry/ziti/zdns.bash] failed to load: failed to initialize ziti
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:715 load_ziti_async() attempting to load ziti instance from file[/opt/netfoundry/ziti/registration.jwt]
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:721 load_ziti_async() loading ziti instance from /opt/netfoundry/ziti/registration.jwt
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO ziti_log_set_level set log level: ziti_log_lvl=3 &ziti_log_lvl = 0x575d2c
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR dk-c-src/library/ziti.c:212 ziti_init_opts() dk-c-src/library/ziti.c:184 - load_config(options->config, &cfg) => -13 (Configuration is invalid)
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR programs/ziti-edge-tunnel/ziti-edge-tunnel.c:632 load_id_cb() identity[/opt/netfoundry/ziti/registration.jwt] failed to load: failed to initialize ziti
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:715 load_ziti_async() attempting to load ziti instance from file[/opt/netfoundry/ziti/output.json]
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:721 load_ziti_async() loading ziti instance from /opt/netfoundry/ziti/output.json
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO ziti_log_set_level set log level: ziti_log_lvl=3 &ziti_log_lvl = 0x575d2c
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO programs/ziti-edge-tunnel/ziti-edge-tunnel.c:630 load_id_cb() identity[/opt/netfoundry/ziti/output.json] loaded
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:715 load_ziti_async() attempting to load ziti instance from file[/opt/netfoundry/ziti/LVT-PRODUSA PenState-StateCollegePA TDC04748 ZLE-01.jwt]
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO lib/ziti-tunnel-cbs/ziti_tunnel_ctrl.c:721 load_ziti_async() loading ziti instance from /opt/netfoundry/ziti/LVT-PRODUSA PenState-StateCollegePA TDC04748 ZLE-01.jwt
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] INFO ziti_log_set_level set log level: ziti_log_lvl=3 &ziti_log_lvl = 0x575d2c
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR dk-c-src/library/ziti.c:212 ziti_init_opts() dk-c-src/library/ziti.c:184 - load_config(options->config, &cfg) => -13 (Configuration is invalid)
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 0.062] ERROR programs/ziti-edge-tunnel/ziti-edge-tunnel.c:632 load_id_cb() identity[/opt/netfoundry/ziti/LVT-PRODUSA PenState-StateCollegePA TDC04748 ZLE-01.jwt] failed to load: failed to initialize ziti
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 1.572] INFO dk-c-src/library/ziti.c:364 ziti_init_async() ztx[0] Ziti C SDK version 0.26.12 @86d3ead(HEAD) starting at (2022-11-11T20:44:47.774)
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 1.572] INFO dk-c-src/library/ziti.c:365 ziti_init_async() ztx[0] Loading from config[/opt/netfoundry/ziti/output.json] controller[https://18.204.45.166:443]
Nov 11 20:44:47 LVT-RPI-EDGE-V311 sh[15324]: [ 1.572] WARN dk-c-src/library/ziti.c:721 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://18.204.45.166:443] api_session_status[0] api_session_expired[TRUE]
Nov 11 21:48:14 LVT-RPI-EDGE-V311 sh[16025]: [ 1.402] ERROR dk-c-src/library/channel.c:244 check_connecting_state() ch[0] state check: timer not active!Hi @William_Penrod, We will look into the logs and get back to you. Thank you!
Hi @William_Penrod ,
I can see that the config is not loading and it says that is invalid. Is that the config file/opt/netfoundry/ziti/output.json? at this point, I am just trying to understand why would that happen.
What is your setup, i.e. host, os, etc? Also, what is the version of your zitt-edge-tunnel?
FYI, I see this log.
INFO dk-c-src/library/ziti.c:364 ziti_init_async() ztx[0] Ziti C SDK version 0.26.12 @86d3ead(HEAD) starting at (2022-11-11T20:44:47.774)
The csdk version is 0.26.12. This was released last year. The latest one is 0.30.8.
Just to sync everyone up here. They are using Raspbian Jessie/32bit, which lags behind the required GLIBC that ZET requires to run. In order to use any newer version of ZET, a special procedure must be used involving downloading a precompiled GLIBC that does not replace the one that comes with the OS. Patchelf is used to redirect the linking in ZET to the non-standard location for the updated GLIBC.
Also note, the deb/apt install methodology will not support this version of the OS/Arch, thus it cannot be used. Once the image is updated to Bullseye/64bit, the deb/apt installer can pick up and manage forward per the documentation mentioned by Suren.
This is only informational, it does not have a material affect on the connectivity questions being discussed.
Thanks @NicFragale for the context.
@William_Penrod Can you let us know how you mange the binary, i.e using systemd service files? Perhaps something is not initialized, when the binary is run.
Can you also, enable debug level 5, perhaps we may be able to see more details of what is happening at boot up time?
Does this have any known drawbacks? Did you try compiling ziti-edge-tunnel on Jessie? I’m wondering if that would be better than patching libc because it seems like that would introduce some undesirable variability to the rest of the OS.