[illumos-gate] 13356 syslog(3c) should not open syslog_door without LOG_CONS -- David Hanisch <titanic⊙dd>

jbk had anymore time to look at the viona <-> sockfs not returning the buffer issue?

I ran into it again today when doing an scp

no i haven't

associative line: can it be similar to MTU mismatch problems?

we had some back in the day of modems in data path of the cross-country links, maybe windowing didn't work, maybe some VPN was unaccounted for, but packets some 4 bytes under ethernet maximum worked and full-size did not or intermittently

like you can spend half an hour in an SSH session and then a big directory listing kills it

hm, or VLAN

scp often stalled at 128KB

jimklimov: complete or for a moment?

this problem is something with either tcp or sockfs + mac loopback path

jbk ack

basically packets from viona->non-viona zone on the same box get queued up in the netstack of the non-viona zone

since they're loaned from viona

once viona runs out of available descriptors, tx from the bhyve instance stops

IIRC when scp stalled like that, it was done for

was long time ago, memory is faded :)

maybe it somehow came around after some minutes, unless tcp timeout struck in

I think the generic solution was, if VPN was involved, to just make the VPN interface MTU small, like 1-1.2kb to surely pass, and it became stable

yeah, i could kinda of reproduce it with scp.. and after a while some timer appears to kick in which unclogs things.. but for non-scp stuff, that doesn't seem to happen

Don't you just need a simple TCP server in the zone which maxes out the socket buffer size and doesn't read anything that it gets sent?

If it's a lending issue

well the problem is there's supposed to be a 'please stop sending me stuff' signal that appears to get ignored

which if that was working, i think would at least stop things before the senders descriptors were exhausted, allowing other TX traffic to proceed

there's also probably an argument to be made that when stuff is lent, there should be a way for the receiver to know and decide 'hey it's going to be a while, let me copy it and release the loaned resource'.. though how to determine when to do that gets tricky

we (amusingly) have some old code (I think related to zero copy) to deal with that on the outbound side, but not for receive

it's a pretty sticky problem, IMO

since normally there's no impact for those long lending times

I'm not sure anything besides viona would benefit from a system for copying those buffers after a spell

Could we just stop lending once you've leant half of the available resources

we could

but it's still visible from the guest

if I ask a NIC to send a packet, and it doesn't for 30 seconds or more...

Yeah but at least it wouldn't entirely stall

With most NIC drivers we reserve an upfront amount of descriptors for loaning, fwiw.

And fall back to alloc/bcopy when we run out.

LeftWing: And there's still the issue of VM shutdown

I guess you can't just leave the relevant pages around with the current architecture

definitely not today

even if you could, teardown of viona gets a little weird

closing the rings down with outstand mblks which expect to make transmit-complete notification callbacks would be a little grimy

*outstanding

one would certainly need to take exquisite care to not leak anything (or use-after-free) if mblks were allowed to be left hanging like that

grimy subsystems is my second middle name. I suspect we can figure out a way to make it work.

well, I think a lot of that grime woudl fall to viona

wouldn't shock me if the ultimate fix was to just bcopy in the case of loopback :-/

we already had to undo the checksum offload for loopback

*checksum ellision

Yes, but I think that'll change over time.

There's a big difference between opt-in versus opt-out.

perhaps!

I mean, there is a long history of IP opt-in already for this stuff.

So far the pkgsrc cache thing in a pkgbuild zone triggers it and from what I can tell in today's case... a cherrypy webserver after sending it a few requests.

Killing the zone fixed the issue, killing just the bhyve vm... make it unable to boot again, probably as hinted above because the viona thingy is not torn down yet

I'd defiantly personally

Opt for less performance over not having to deal with this.

Max I waited was about and hour and it didn't continue... so I assume some magic timeout does not fix it in all casss

I wonder if we could first add a tunable that forces copies across the board here

And you could tune it on

Without hopefully a great deal of upheaval

We should be correct before we are fast

Is there a bug filed for the hang/stall

i thought so.. let me see if i can find it