Humor Random Comments Thread

[jedidia]

A single race condition that can lock up the entire system? Wow, that's rough...

 

[Urwumpe]

A single race condition that can lock up the entire system? Wow, that's rough...

Yes. Its especially terrible that nobody caught this issue in the 15 years of development before we came into the project. :lol:


They had all known that some numbers don't add up. They never expected that the numbers could add up so badly, that the system can never recover.



As long as only 2 jobs per second arrived, all was fine. But at 100 jobs per second, it took just 5 hours until the system is blocked.

 

[jedidia]

As long as only 2 jobs per second arrived, all was fine. But at 100 jobs per second, it took just 5 hours until the system is blocked.

And from what you're describing, I assume that's not due to lack of power, but simply because some jobs are never dequeued and pile up. Yeah, there should really always be a ttl on jobs waiting for a lock.
*thinks of code he wrote during the last month*
Uhm... If you'll excuse me, I just realised there's something I needed to do... :leaving:

 

[Urwumpe]

And from what you're describing, I assume that's not due to lack of power, but simply because some jobs are never dequeued and pile up. Yeah, there should really always be a ttl on jobs waiting for a lock.
*thinks of code he wrote during the last month*
Uhm... If you'll excuse me, I just realised there's something I needed to do... :leaving:

Much worse: They are properly dequeued. :lol:


But the count is not properly updated. Instead of counting the number of entries in the job table, somebody wrote the code like that:



When accepting a new job:
1. Fetch current number of jobs for this server from DB
2. Add one.
3. Store current number of jobs for this server in DB


:rofl:


Guess how fast and reliable the software was after just maintaining a table of job assignments...

And yes - having multiple watch dog timers in your software is a smart idea.

 

[Notebook]

You will have to explain all above in slow language before I even start smiling!

N.

 

[Urwumpe]

You will have to explain all above in slow language before I even start smiling!

N.

Well, in slow technology. :lol:


Imagine a classic mechanic score board, just replacing numbers by taking one off and hanging the next score there.



Right now it shows 4.



You have three parties of chess happening right now. Everytime a game is over, the referee should note the new score on a paper and pass that paper to the person maintaining the score board.


As long there is only one match ending until the next update of the score, all is fine.


Now, all three end at the same time:


First referee reads 4, writes 5 on the paper and passes it to the scorer.
Second referee reads 4, writes 5 on the paper and passes it to the scorer.
Third referee reads 4...




As you can see, not only are computers very fast, but stupid, but also the score is wrong then: Instead of 7, as expected, you read 5.

 

[Notebook]

Understood, sort of smiling in a don't quite get the joke yet, but understand a bit more than I did!

If I may give a similar type anecdote.
I worked in a few tv transmission areas, mostly without much trouble.
One channel had a fairly standard afternoon format, programs, breaks, with a live studio doing the ins and outs of the breaks.
There was a fixed event at 19:00 every week-day, a family film. It was a children channel.

There were two of us in the area, the Transmission Controller, who didn't have to be technical, and the tame engineer who was.

All went well till a new Studio Director arrived and decided that the Studio was more important than the programmes. There were nominal times for the studio content, but as it couldn't be scheduled to the second, they became "Live Events" in the playlist/schedule.
The automation held during this time, and went back to schedule when the TC pressed the "Take" button.
Our new director took to adding time when he got an interesting caller( it was mostly school children talking to the two presenters), he wouldn't give it back though on the next break.
Eventually the automation got fed up with this when it realised there was too much playout material till the fixed event at 19:00 and turned the rest of the playlist a nice shade of red.
An argument ensued on talk-back as to where the extra(less) 30 seconds was going to come from. He insisted we lose some of the next break material, he wasn't going to lose any studio time.

We shouldn't have got into this situation, just goes to show what happens when you help someone out.
Dan the Man(TC) eventually lost patience with him and had me do a 30 second count to where the break will be ending, and chopped off the studio.
This all happened in a two-minute break, and very untidy it looked.
Usual post-mortem, nothing solved. He didn't try it again when myself or Dan were on shift, but he did try and bully others. Didn't last long.

I'm imagining something similar is going on in your database races...?

N.

 

[Urwumpe]

I'm imagining something similar is going on in your database races...?

N.

Not quite - but a similar phenomena exists in IT as well. :lol: You pretty much described the Dining philosophers problem with TV technology.



https://en.wikipedia.org/wiki/Dining_philosophers_problem

 

[Notebook]

Fascinating, didn't know about that.

Our problem was self inflicted, never mind the human element. Scheduling allocated more time to the studio than was necessary. the whole idea(from transmission side) was that the studio would take up the slack, and the last break before the movie would be the ultimate filler.
Didn't work for this guy, he thought he was the next Cecil B. DeMille(Spielberg of his day).
Had another run in with him when he wanted to lose subtitles on program ends. Programme format at that time of day was a reprise of the plot, and an opportunity to do a few gags, so dialogue ran right up to 10 seconds from end.
The subtitle generator took time-code from whatever source was on air, obviously a studio only has time of day as its output. Result was if we did as he wanted, leave a studio o/p squeezed in the corner, corrupted t/c and the subtitles would freeze.
He obviously had never watched the channel output, and gave a very sneery comment on the ending subtitles "What's it going to read, great end music?". He lost that one.

N.

 

[Urwumpe]

Our problem was self inflicted, never mind the human element.

In my experience, 99% of the technological problems are actually caused by human element. Somewhere, sometime, somebody decided to take risks, that he was either not aware of or decided to take.



Or simply decided to make this the problem of somebody else.



In one of my first space flight lectures, the professor reminded us to define exactly which stage has which part of the separation systems. Without, every team will think the other stage will handle the separation. :lol:

 

[Notebook]

I found it odd that the company we were handling the transmission for were quite laid back about some procedures. They were very protective about copyright and the use of their logos and trademark.
I think it was because we were in with it from the beginning of their transmissions in the UK. It grew from just the one channel to multi-and international channels.
I do remember getting a call from Sven in Helsinki telling me we had all the wrong languages on our Scandinavian subtitles. As I can't read Norwegian, Danish or Swedish, a bit of a problem there.
Another problem was the satellite footprint didn't cover our latitude, that why they did a deal with Sven to keep an eye on it. Passed that one on to SKY as they were handling subtitling then

I kept with it for a few more years till they went tapeless. I think at one point we had 15 tape machines on hire in the tapeless areas...
Ended up with Sony Broadcast, Harris Automation and Grass Valley and the client nearly in litigation. Took early retirement and did freelance stuff.

N.

 

[PhantomCruiser]

Watts Bar Unit 2 in mode 4.

The end is near.

 

[Linguofreak]

I'm imagining something similar is going on in your database races...?

N.

A better way of explaining what he's getting at is that you have a shop that takes orders by mail. There is a large billboard at the front with a large banner pinned to it that shows the current number of orders that have been received but not processed. Nobody is allowed to go home if it shows a number higher than zero, and if it shows zero at any time after noon, everybody is required to go home for the day.

There are three secretaries each checking in incoming orders. Whenever a secretary finishes checking in an order, while still at their desk, they take a new banner out of a box, look at the billboard, add one to the number already pinned to the billboard, and write it on their banner. They then walk up to the billboard, throw away the old banner, and pin up the new one. There are three box packers that do the same when they finish processing and shipping an order, except that they subtract one from the number currently on the billboard.

The problem is that, since everybody writes the new banner at their desk, the banner you looked at to add one to might not be the same banner that you replaced when you finally got up front. So if several secretaries are all checking in new orders at the same time, they might all see that the billboard indicates 4 current jobs, each write "5" on their banner, and then have the first one replace the old "4" banner with "5", the second one replace the first one's "5" banner with a new "5" banner, and so on. So if there were three new jobs, the job count should read 7, but will actually read 5. If the box packers complete their work on the corresponding jobs at a pace such that they don't all reach the "update the billboard" step at once, then the subtraction of the completed jobs will proceed normally, and the job count will reach zero before the actual number of pending jobs is zero, and, if it's after noon, everyone will go home with work still pending.

It could also happen the other way, to where new jobs are added normally, but several completed ones step on each other at the update billboard step. Then the shop runs out of work to do before the count reaches zero, so nobody can go home because The Rules say that the count has to be zero before anybody can leave, and everybody has to stay overnight and hope for an offsetting error the next day.

This is, of course, a completely daft way to run a shop, but the way computers work makes it easier to run into such problems than in the physical world, so programmers have to be trained to spot such problems and build safeguards against them.

And actually, with servers, the "running out of work before reaching a count of zero" scenario is often *less* troublesome than the "reaching zero before running out of work" scenario, because if the program tries to get the next work item when there is no such thing, it will, worst case, try to access memory that doesn't exist and crash, and then the operating system will restart it, and everything is fine and dandy. On the other hand, if there are jobs waiting and the program thinks there aren't, then each of those jobs will take up memory. Not alot, but if you've got 100 jobs per second coming in, and half of them don't update the count properly (and remember, the heavier your load, the more jobs will try to update the count at the same time), then even a small amount of memory (10s of kB) per job can use up 10s of GiB in five hours. And when there's not enough memory, the OS will start using the hard disk to pretend it has more memory, but hard disks are s...l...o...w... .

And as using disk to emulate RAM slows the system down, the process of reading the job count, adding one, and writing the job count back slows down, so the window for jobs to step on each other updating the count gets wider, which means a higher percentage of jobs fails to update the count correctly, more memory gets tied up, more disk is set to the task of pretending to be memory, so the system gets slower.

And *then*, 100,000 customers that have been trying to get to your website for the last 10 minutes all have their browsers tell them that the page load timed out, so they all hit refresh, and now you're getting 200 requests per second, 100 from new customers, and a hundred from all the waiting customers trying to refresh the page.

And then your tech support line gets a call spike and suddenly wait times for tech support are going up, and pretty soon grumpy customers that have been listening to hold music for an hour are chewing out underpaid agents that have had back-to-back calls with nothing but grumpy customers for an hour, and pretty soon emotionally spent agents are snapping back at customers and avoiding calls, and then a customer makes a recording of his conversation with an exceptionally grumpy agent and parts it to social media and/or the press. And then the board of directors is asking pointed questions of middle management, and middle management deflects the issue to the software engineers, and then Urwumpe says "we opened a ticket on this two years ago".

But you can see from the above that Urwumpe got one thing wrong. He was too modest:

The destructive yield of emails like that is measured not in gigatons, but in units like giga-foe and millions of solar masses * c^2.

Somewhere in the most uninhabited reaches of the Andromeda Galaxy, underneath the outback of some uninhabited desert planet, or at any rate, far away from Wolfsburg, a German middle manager is digging with inhuman speed towards the core of the planet, crawling into the deepest, darkest hole he can until the storm passes.

---------- Post added at 08:26 ---------- Previous post was at 08:15 ----------

The end is near.

"Get a life you fanatics!"

SCREEEEEEEEEEECH!

SPLASH!

"I told you the sign should have said 'the bridge is out'!"

 

[Urwumpe]

Somewhere in the most uninhabited reaches of the Andromeda Galaxy, underneath the outback of some uninhabited desert planet, or at any rate, far away from Wolfsburg, a German middle manager is digging with inhuman speed towards the core of the planet, crawling into the deepest, darkest hole he can until the storm passes.

We suspect that managing this project is already the divine punishment there. Not getting away and to a more prestigious and less political project (many departments are stakeholders, so your are under constant flak if you are not doing what one of those wants, even when all others disagree.) is sure hell.



As far as I can tell, we developers managed to leave, though it wasn't easy at all, while the managers are still there. I know of one who tries to leave the project for the past 5 years without success. When we had been new in the project, we joked about the key persons leaving the project only in a casket. Today, it is not funny anymore.

And the good thing is, nobody will ever really know about the project outside its organisational context. No press, no angry users, no torches and forks. Even if maybe half of the planet is affected when the system failed, even the experienced users did not even know that this software was existing. It was hidden deep inside the backend of many other services, with only very few people knowing about its function. Just imagine that it took a small bug to find out about all connected systems: We accidentially corrected a wrong return code and suddenly found all systems, that directly connected to the webservice and had fixed the bug on their end.

 

[Linguofreak]

And the good thing is, nobody will ever really know about the project outside its organisational context. No press, no angry users, no torches and forks. Even if maybe half of the planet is affected when the system failed, even the experienced users did not even know that this software was existing. It was hidden deep inside the backend of many other services, with only very few people knowing about its function. Just imagine that it took a small bug to find out about all connected systems: We accidentially corrected a wrong return code and suddenly found all systems, that directly connected to the webservice and had fixed the bug on their end.

Well, yeah, I went for the absolute worst-case scenario to drive home the point about what happens when unserviced jobs start to accumulate in memory until the server thrashes itself to death on swap. I'm not sure every step in that chain has ever happened all together, but certainly large chunks of it have. But still, even if the public doesn't know what you do, if the failure of your product leads to somebody having a PR disaster, even if that organization doesn't know who you are or what you do, they'll give their vendors grief, who will in turn give their vendors grief, until somebody who knows what you do gives you grief (and you get to bring out the 2 year old ticket). Crap, after hitting the proverbial fan, always rolls down the proverbial hill.

 

[Notebook]

Sounds like the scenarios that happened in the last place I worked.
Luckily I didn't work for any of the main contractors I mentioned above, we were just the operational side of the transmissions.
From the presentations we had from the system architect, the design would provide four dedicated multi-channel presentation areas, a spare/training area. Would be able to handle
all ingest requirements(transferring from tape to server), provide quality-control at various points in the chain. Edit suites would be able to work on "near-live material"
That raised a few eye-brows. Scheduling would be able to modify live play-out.
It was as well built as any installation I'd seen, unfortunately it worked about 90% of the time, and some parts didn't work at all.

I got fed up after 18 months and quit, kept in touch with my old workmates. It nearly got all fixed sometime later, but by then HD was coming and a lot of the equipment got replaced and they went back to one channel gets one area.

Shame really, would have been quite amazing to watch if it had worked.

One of the early video servers made by Tektronix.
https://www.broadcaststore.com/pdf/model/17257/Profile family.pdf

If you scroll down to page 17, it has a brief description of its advantages.
Its the last paragraph that always get me, "you can start playing a clip while its still being recorded" and repeats that later.
Never understood why you would want to put something to air when its not finished?

N.

 

[Notebook]

I'm afraid I got completely the wrong idea about this:

https://www.instructables.com/id/Iron-Man-Stool/?utm_source=newsletter&utm_medium=email

N.

 

[Urwumpe]

Well, yeah, I went for the absolute worst-case scenario to drive home the point about what happens when unserviced jobs start to accumulate in memory until the server thrashes itself to death on swap. I'm not sure every step in that chain has ever happened all together, but certainly large chunks of it have. But still, even if the public doesn't know what you do, if the failure of your product leads to somebody having a PR disaster, even if that organization doesn't know who you are or what you do, they'll give their vendors grief, who will in turn give their vendors grief, until somebody who knows what you do gives you grief (and you get to bring out the 2 year old ticket). Crap, after hitting the proverbial fan, always rolls down the proverbial hill.

Well, as I said: Until there is an investigation finding out that this software existed and who developed it, the bug was likely fixed already in the past. :lol: Not sure if this still applies...



In a few weeks, the first customer user certificates will expire... guess who will have some (microwave) popcorn at work on that day. We had written how to create them properly into the operations manual, but I did not mention our extra services that the customer did not want to pay....

---------- Post added at 11:18 ---------- Previous post was at 00:27 ----------

How you can prove as nationalist political party, that you have no idea what goes in in your immediate neighborhood:

Hang two posters in a short road in an industrial area. One championing small businesses in favor of big international companies. The other ranting against ethnic diversity.

There are two major businesses in that road and a collector of old construction machines.
Both companies in that road owned by big international parent companies (One Indian, one French)
Both had been considered large enterprises for the past 15 years.
Both companies work worldwide in the IT business (One does hardware, the other software and consulting)
Both companies have a very high diversity among their employees.

Location, location, location... :facepalm::lol:

 

[jedidia]

So... How do you something up as good as Game of thrones season 8? Like, even if you were trying, how?

 

[Urwumpe]

So... How do you something up as good as Game of thrones season 8? Like, even if you were trying, how?

By making the long expected main battle of the series look like a cheap ripoff of LOTR?


By violating the Chekovs Gun rule too often?