[illumos-gate] 13389 After persistent L2ARC import, cache device has constant 8KB/sec load -- George Amanakis <gamanakis⊙gc>

Has anyobody a guide on how to start a IPS or rather another SMF instance of a service? I am running into the too divergent from Oracle docs problem.

it depends a bit on the service itself -- IIRC, some services are defined as 'one instance only'

so i guess, what service are you trying to do that to?

jbk application/pkg/server it's designed to have multiple instances.

And I need to overwrite two properties. port and root folder

Thats how the current OI hipster repo is installed. But I need to make a new guide how to do it I noticed.

hmm.. the manifest looks like it's setup to allow multiple instances

well i don't know how well i'd trust oracle docs period

Yep :) but I can't for the life of me find the commands to make an inatance

toastersonerson1: So

i once tried to make a local solaris 11 mirror so that all my servers at $JOB-1 could update from that instead of all having to pull off the internet (esp. given how unreliable pkg.oracle.com was)

What I do is

Oracle's SMF is extremely differnt than ours now.

and their docs _never_ worked

svccfg -s pkg/server

And then you can "create someinstance" and "select someinstance"

"setprop pkg/inst_root = astring: /ws/public_repo/repo"

"setprop pkg/port = count: 9000"

etc

quit out of svccfg

svcadm refresh pkg/server:someinstance

svcadm enable !$

If you want to add more properties in future you can get back to it via: svccfg -s pkg/server:someinstance

LeftWing (IRC): ok thats what i though. Are you sure it's create? Thats not documented in the manpage but add is.

You're right, sorry, add

But add made something I could not do setprop

actually

Ah so it is add

wait

Yes it is add

I will fix my notres

notes

ok then lets try again based on your notes this time :)

Also I suspect I missed the "addpg pkg application" step that should be after you select the instance

(to add the property group you're going to setprop into)

Roughly: add instance, select instance, addpg pkg application, setprop pkg/...

Then "svcadm refresh" takes a snapshot of all the properties you're editing, and that's what the service will use the next time it starts

LeftWing (IRC): Yep addpg is needed aswell :)

You may also want to set pkg/proxy_base if you're putting it behind a reverse proxy

And possibly pkg/address to bind it to localhost in that case too

LeftWing (IRC): ah proxy_base I need to lookup if I need that.

But thanks that worked :)

I think if you don't set it, sometimes it puts links to the wrong place in the generated HTML

e.g. you'd want to set it to, say, pkg.openindiana.org/hipster or whatever

Yeah we first need to figure out how to migrate before we do that :) first let's see if it can be filled with stuff. and oi-userland can be built on the new build zones.

Anybody made good experiences with certbot BTW?

I've been using github.com/dehydrated-io/dehydrated for a long time

I second dehydrated.

hmmm I am still kind of a caddy fan :) no need to worry about let's encrypt setup when the proxy does it, but for DNS challenge it might be interesting.

I mostly put nginx in front of things

For DNS challenges I've used acmedns a bit: it provides an API where you can insert the challenge records with curl in a hook script

(you delegate a portion of your DNS space to it)

Considering there are more nginx trained people in the OI community aswell I'll go with dehydrated aswell I think. Makes things more maintainable.

Do we have something similar to initifywait?

inotifywait

Nothing native -- what are you trying to do?

reload nginx if a file has been added to a directory

That sounds like a recipe for breaking things haha

Is this for certificate updates?

time to fully implement SMF file dependencies!

(I'm joking)

Time to fully implement SMF!

Leftwing no it's for dynamic PR testing repo servers.

Great art is never finished, you just stop painting

hey watch where you're pointing that fully implement stick, people could get hurt

it could use profile functionality :)

toastersonerson1: If it's for nginx + dehydrated, I just restart the SMF service in the "deploy_cert" hook

LeftWing (IRC): It is but not because of that :)

We want to create many pkg/server instances for each PR and then add those to a reverse proxy as path. only one host

I think there was a pattern thing in nginx but I am also checking options.

You probably want one of arekinath's nginx lua things

oh that sounds like a plan. like extract path from a file on request kind of thing?

I think once you get Lua in there it can do pretty much anything

Ideally you'd just look at the current set of pkg_server instances for ports and so on

yep. publisher;port kind of a deal.

Agnar and I may also end up using only one repo and many publishers.

But one server out of the build of jenkins is the current experiment.

Hmmm that dehydrated thing was enjoyingly painless

the openresty people have been making using nginx-lua progressively more and more annoying lately

nginx is shipping with a (limited) javascript engine now, njs

so they've decided they need to differentiate or something

net result is you now need other bits of openresty, not just nginx + nginx-lua any more, and you need LuaJIT specifically (lua 5.1 is not ok any more)

and of course LuaJIT is annoying on illumos because of the stack being up in high addresses and it using tagged pointers for lightuserdata

so any extensions that try to make a lightuserdata out of something on the stack (looking at you, LPEG) blow up

unified testing repo with PR publishers becomes more and more an interesting option.

toastersonerson1 looks like you’ve got what you need for the depot setup, but there are also some example scripts in the “making of” section of the omnios web site that might be interesting

for

example

andyf (IRC): Thanks I'll have a look.

I think the acmefetch config there is a little out of date, but the rest should be ok.

danmcd: your monitoring vmstat with your server situation why exactly? If it were an OOM panic wouldn't you see evidence of a dump from the kernel?

also, are any of your drives on a sketchy backplane, perhaps?

KungFuJesus: I'd worry more about sketchy backplane but all of the drives are direct-SATA-to-mobo, and you'd think the sketchiness would've manifested earlier in the 6months it's been up, no?

I'm not thinking the backplane is screwing up AHCI traffic - more that a broken trace is causing flakiness over the 12 or 5v rails

but if you're not even using a simple SATA backplane and you're wiring the drives up directly, then that's probably less suspect

any sort of weird data errors taking the machine down I would suspect would manifest as panics to the kernel for the root pool, not random power outages with no evidence of a crash

So what's you're instinct on this? Flaky power supply? Or a bad drive doing really horrible things?

Uggh, s/you're/your/g

danmcd: i just read your e-mail. my experience is that this may be caused by either flaky PSU or a CPU cooler/fan not doing its job properly.

my wife's computer rebooted every now and then, after cooler+fan replacement, it works like a charm.

(one of my servers reboots every 1-2 weeks, i also suspect cooling problems)

I don't suspect cooling as much because it's cold in my basement, the machine has been up for 6mos w/o this kind of failure, and wouldn't doing an illumos-gate build (30mins of full-blown CPU and disk) warm it up?

These are 6-hours-into-a-scrub-it-fails sort of failures. I'm more suspicious of PSU for that reason.

Lemme look at my temp sensors...

not only temp, but also RPM

thermal paste?

i had to re-apply..

7200RPM.

CPU says 26C and we're still scrubbing (but one drive). One drive scrubbing appears to NOT tripthings up.

did you run memtest?

Not yet, plus wouldn't I get a kernel panic if it was bad memory?

depends on the MB I think

ASRock Rack X470D4U.

depends on where the bad memory is

or rather, whats in there

I'd have to power it down and check BIOS for ECC-settings.

yes, memory problems can be very tricky to diagnose

IIRC, some vendors do stuff in their BIOS that can interfere with an OS trying to stay aware of memory issues

(And I can't take this offline w/o warning the whole house. Calendar and routing live there. I can replicate the routing, but not necessarily the calendar. Need to do those sorts of ops on the weekend.)

once had one out of four dimms have a stuck bit and no ecc, that showed all sorts of interesting errors, including zfs errors that propagated all the way to the disk with correct checksums...

rotten bits?

But my failures manifest only one way: spontaneous power-cycles.

probably not memory

yes, the micro dust in dimm slots/pci slots can also be issue

are they resets or power cycles? if resets, could also be memory. if power cycles, then most probably PSU

power-cycles. IPMI *appears* to be unfazed, and no records of resets in IPMI or illumos.

Interesting, IMHO, that it takes sustained IO load of `zpool scrub` to trigger, and that last night's scrub-start reported a disk failure, and that the scrub is still going (and is near completion for the still-online drive).

So PSU seems a good front-runner?

I'm pissed if that's the case. it's got a 10-year warranty and many 5-eggs. I guess the 1-eggs might be more correct?

not even the IPMI logged anything?

Nope!

how does the IPMI survive a power cycle?

That's what I asked!!!

Only one odd IPMI event, with timing not anywhere near the power cycles.

I even ran the camera-on-the-screen in case it was a triple-fault. :)

my instinct is that it's not likely a flaky disk could cause power issues, but likely power issues could cause a disk to be flaky

Recommendations for PSUs that aren't Seasonic?

Not redundant? Seasonic is usually a reliable brand

Most power supplies are really mainly made by a handful of factories, regardless of brand

I like Asrock as a consumer brand, but they have had their fair share of oopsies, if I recall. Namely on the ITX form factor, but they are relatively new to the server board scene

it would not surprise me to find your issue there

this is a stupid question, but...do you have it on a UPS? Are you sure that's not randomly tripping under load?

If not, maybe try the wall power, if your source of delivery is reliable

I run things in my basement and had my fair share of pure annoyance with GFCIs, too, so it could also be the other way around

My experience, apart from my expensive as all hell rackmount power supply, is that he UPS is the weakest link in the chain

you can confirm a suspect UPS, sometimes, by seeing if it has any events listed

but a lot uf UPSes fail in weird and catastrophic ways

for instance, some may not warn properly when the battery is failed, and an intermittent delivery from the wall can cause the UPS's relay to click on and try to draw power over the battery, and fail, silently

_sometimes_ UPSes are more sensitive to line voltage dips than direct computer power supplies. In which case straight from the wall would be fine. Some UPSes that try to constantly regulate voltage can also fail when the battery health starts to dip and it notices a natural voltage sag due to a heavy load

I do have it hooked up to a UPS (which just got a fresh battery change). But this UPS has not been a problem for more than a year w.r.t. power-flickers.

I don't have it hooked up to anything where I can query it, unfortunately.

Well, the scrub (on one drive) finished. I'll run smartctl -x later this evening in case things are so fubar that even doing that will powercycle things.

What is the ls-like command that shows the directories with mount points and such (in SmartOS it shows what's ramdisk)

silly me df -h

jbk was it you telling me about headless loader issue?

you mean the ACPI SPCR stuff?

some system without video getting hung

hrm.. that doesn't ring a bell

though the past couple of months have been hectic

ok. just that I did fix similar issue with fbsd today and I'm quite sure the same issue is with our version:D

but, the good thing is, it also means, we can have ACPI SPCR work to go on...

[illumos-gate] 13571 cron: symbol 'err' has differing types -- Toomas Soome <tsoome⊙mc>

danmcd: in my experience, most sub $500 UPSes are kind of crap with regard to how they behave. Can you see if there are any events? Does it have a self test mode or can you try putting an especially large load on it to see if it shuts off?

I don't have it hooked up to anything, so it might be hard to get events from it. And what's weird is that this very UPS has been in service longer than the 6month lifetime of the server. Unless it's wearing out (which it could, I admit), I'm not sure it's to blame.

KungFuJesus: the last cheap UPS I bought was _less_ reliable than power straight out of the wall

True here, too

danmcd: I would try to rule out the UPS to be honest, it seems the most likely suspect

before I file these two tickets... currently a _lot_ of kernel SDT probes are missing their CTF argument information -- this is because every module must also manually define all of that in usr/src/uts/dtrace/sdt_subr.c (and it appears almost no one is aware of this)

i'd like to file a couple of tickets: one to make the SDT probe data more modular (i.e. each module defines it's own SDT probe data and registers w/ the SDT provider on attachment) and a second ticket for the missing SDT arguments

without it, you have to cast argN to the correct type (which requires finding the probe definition in the source), which just makes it more error prone

ah.. you know, that explains a lot

heh yeah

I was playing with the probes in pageout, and had to cast everything

we noticed it when dtrace -vl wasn't showing any arguments for some SDT probes

and tried to figure out why

and finally tracked it down

that so many (e.g. _every_ zfs SDT probe) suffers from this suggests no one is really aware

i would much rather be able to do arg[0]->foo and have dtrace spit out an error if i get it wrong than possibly show garbage or throw less useful errors

(I think it's separate from any stability classification -- you can still define the arguments and classify them as 'private' to tell everyone 'these may change at any time, use at your own risk')

/!\ this chat has moved to irc.crimeircd.net #0 /!\

/!\ this chat has moved to irc.crimeircd.net #0 /!\

/!\ this chat has moved to irc.crimeircd.net #0 /!\

/!\ this chat has moved to irc.crimeircd.net #0 /!\

/!\ this chat has moved to irc.crimeircd.net #0 /!\

/!\ this chat has moved to irc.crimeircd.net #0 /!\

oh yes, crime irc network, totally safe I'm sure

PRsE68L2UCUbc8yH93baUVR2xKj8DHJh3CcxotO6S8v8SS6RzbkG9PrTa19B0t2zJVwbwmSuX2zHw5brrujgxlCrYIf4ybq0Zsk5niOEgo4FbvD9Cj6XmX6wHXm4AgLIQmCNEzLMuzViIlIZJf6WXfpIF

RYwTnUFqbZL2aqdZVpq8Yxb50WYymo0T4LfDPwRYnz59RCYlQWmr363UgGH470eeLzO4OR6D0IAf2nIVQVgTTQz9v9cnEYH3umkeqh1m2H8H7me48k8ZLMu7lBWg3uSoDPiPuIcZ5aK53N5ncFzQl7WuY

ztl9a7xiP4NNYTpnxdB6Z5xwGsAZGMg0PyXeIDGQTTDj1ZAFxI1VhxaYxaMxdwUHNqpQfqQjY85emJD0U7GQiQ1FkaPIfRhPpNVj8gnL4Bagtku3VmWl72R9jIZdcEignll7ZrLzUoug3AGZFtF6eonHH

Nj8I2iC4LWYxxYNbh6icH0yAKEtwJ6j2ZzQ7p3cWM2JohA9liLph1cmxLtj5NR9VS0YeX7nhOtMdaI7WC1pOiTEauerASpad0M71CwbD1v7jI5WaxdRop9DtYEWYIVF5DLZOrQapFEX4mPnKxkS46HmRr

JwouATN2Er21aYWmfHwhWDGEGHSUETCKuO66iTI75xKrDnN7ww8oLrcrSqzAitMmYL2IZuR44oDleBXT17UpA1jrt997oYKbTZeNQ0KWiw68dG75ZmbzN1JY8pPFSekJyOFWOGhGSzO3vYcJYRdjxaVwg

anyone have any idea what in the world ITM stands for?

richlowe: in what context ?

geniconvtbl

sorry :(

if it's gen iconv

the first asspociation was international trade mark

before I read the context I was thinking of international trans meat but just didn't seem to add up ,)