I have an issue which I have had a tough time getting answers for, even with my best attempt at Google-Fu. My company infrastructure uses serial port (picocom) for several things, and these serial connections occur over ssh connection. Everything works fine without the ziti-edge-tunnel involved, but once we introduced it to some users, I have been hearing complaints of the serial connection terminating or hanging. I have some theories (mostly speculation) that it could be due to the buffer limit not being reached, or that it is something to do with ssh encryption (specifically the NetFoundry service config), but these are just hunches. Has anyone else seen a similar issue? If so, did you fix it? Thanks.
I’d be interested in more detail about how this is configured. I’m not overly familiar with picocom, but have used similar technologies in the past.
Sure. Basically, imagine it like this:
Originally, the end user connected to the remote server via ssh to a public IP. Once on the server, the user runs picocom, specifically picocom /dev/blah -b 11520 (where -b is baud rate). No complaints.
Now we introduce NetFoundry into the equation. We instead use a NetFoundry service hosted on the remote server (also running ziti-edge-tunnel) to intercept the ssh connection. The interesting part is, ssh works fine on its own. Only once the user connects to the serial port is the issue noticed. My coworker suggested that it could be due to something with how ssh handles encryption, where the buffer limit (or rate?) is not reached by the serial connection (due to the data being serial), and the serial connection hangs & eventually terminates. Many users in many different geographic locations have complained about this. Unfortunately, I can’t even test turning off the NetFoundry ssh service encryption (it won’t allow me to turn it off), so I’m a bit stumped. I am unsure whether the issue lies in the ssh service running on the ubuntu server, the ssh NetFoundry service configuration, the picocom serial connection, or some combination of all or some of these.
Is there any other detail you’re curious about?
Can you open a support ticket via the console? I’d like to get some logs from the system and follow up more than we should do publicly.
In the nfconsole, the question mark icon in the lower left will give you a menu and let you enter a ticket from there. That will capture some information and submit it to us as part of the ticket. If you could also point out specific service names that experience this problem, we can do some digging. Lastly, once you have the ticket, if you could get any logs from the sysmtes involved and attach them, that would be great. Depending on how it is running, you may need to get them via “journalctl -u ziti-edge-tunnel” or similar CLI, if you’re not directing the logs specifically to a file.
I spent some time on Friday trying to recreate the issue, to no avail. I will continue to do so today, and when I have recreated this issue, I will upload the logs as you said.
Okay, I recreated the issue. The only initial thing I noticed in the logs was this error May 22 13:49:43 eastport ziti-edge-tunnel[99814]: (99814)[ 925295.068] WARN ziti-sdk:conn_bridge.c:300 on_ziti_data() br[0.1430] closing bridge due to error: -23(connection is closed).
I am submitting a ticket now.