Rendered at 15:51:29 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mikewarot 2 hours ago [-]
Every since the OPM hack of 2015, I've been apparent to me that my former field of IT administration has lost the plot. Nobody knows what a data diode is, or why you would use one. Systems that should clearly be air-gapped aren't.
While it's easy to lay this at the feet of AI getting better at hacking. I see it as an primarily an IT issue. We've collectively ignored the lessons of history, and made do with patch jobs over poorly chosen operating systems instead.
--
We need air gaps, data diodes, and capability based operating systems. Now that I'm retired, when I have the energy, I'm working on the data diode part.
This weeks lesson for me, personally, as I try to build an open source data diode, is that the Waveshare RP2350-ETH is a horrible choice for a proxy/data source/sink, as the CH9120 ethernet interface can't do promiscuous mode. It might still be sufficient to build a data diode that can mirror a website, with << $50 component cost. Time will tell.
jandrewrogers 29 minutes ago [-]
In the last few years I’ve seen multiple startups describe normal systems running on AWS as “air-gapped” because there is a firewall. LARPers have diluted several terms that had specific meaning in a system isolation and security context to the point that I can no longer trust when people describe their systems using them.
There are far too many unserious people representing our industry.
m463 16 minutes ago [-]
> Systems that should clearly be air-gapped aren't
Personally I think refrigerators, microwaves, washing machines, vacuum cleaners, ...
I think at some point people just give up or the (bad) idea gets normalized.
Then at work, they're not surprised when some other system is connected for convenience...
EvanAnderson 1 hours ago [-]
> Systems that should clearly be air-gapped aren't.
I'd take systems that were permitted to have filtered egress at this point. The lion's share of my work is in networking, so network segmentation is the "hammer" I pick up first.
Vendors gnash teeth and complain when I ask for their app's dependencies on hosted APIs and off-site resources. In the environments where I'm mandated to maintain FBI CJIS compliance I can still hold vendors accountable and get what I want. It's pretty much a lost battle in every other environment and unfiltered egress to the Internet from servers is just expected.
That's not even to get into the topic of communication flow within an application. >sigh<
wat10000 52 minutes ago [-]
I would argue that the industry is learning the lessons of history quite well. The problem is that the lessons of history are that data breaches aren't really a problem for the breached organization. You get some bad press, maybe pay a pittance for identity theft monitoring, and then move on with your business. Why put effort into preventing something so inconsequential?
daveguy 1 hours ago [-]
I'm curious what function a data diode (unidirectional network) would provide besides sensor data/mirroring/replication. Once you do anything that requires packet confirmation (TCP) you open yourself to OSI layer 4 security risks. Seems like mirroring/replication would need some feedback and then is it just sneakernet restore? Or another diode out from the replication for specific processing? If the same control system has access to both diodes then it's not a unidirectional system anymore. Is a data diode more of a pseudo-unidirection where it is enforced above TCP?
I completely agree that IT admin could be a lot more secure by design. Combined with better interfaces for responsible configuration.
dezgeg 40 minutes ago [-]
Not to mention how any sort of TLS key exchange is supposed to happen with an unidirectional network.
mikewarot 15 minutes ago [-]
Nothing is supposed to ingress, that's the point. You'd do a TLS exchange with the proxy on the outside of the protected network. Then proceed as normal.
>The Work Number is an Equifax (who famously had a massive data breach a few years back) owned database with employment and pay information.
>You can create a login on theworknumber .com and pull your report. Mine is seventy four pages and had info from every company I've worked at in the last 13 years including every paycheck I had received with the exact dollar amount (both net and gross) and hours worked. It has employment start/end dates, termination reason, details about withholdings, benefit enrollment, union affiliation, etc etc etc.
>This data is sourced directly from the HR platform your employees use (ADP, Rippling, Gusto, iSolved, etc). Equifax sells your data for things like employment background checks.
>You cannot have your data removed from The Work Number. You can freeze your report (much like a credit report) but if a potential employer cannot access your frozen report that may disqualify you.
>Why YSK: If you're interviewing for a job you can be at a significant disadvantage especially for things like salary negotiations because they can literally see how much you make including your most recent paycheck. Creditors also can access these reports.
>This is the biggest privacy violation I have ever seen and almost nobody is even aware of it. Certainly none of us consented to having our employment and income data harvested and sold, especially since we get nothing in return. At a minimum, people should know about how this data can impact you.
>Edit: The Work Number is primarily for the US, but there are similar services for other countries.
tomrod 4 hours ago [-]
I checked my data. It is either 100% wrong or misleading. Start and end dates, compensation amounts, benefits. Per the file, I've worked 4 years out of 20 in my career. As an employer I have never used them and I worry for any employer who does.
ciupicri 6 hours ago [-]
From reddit:
> In my country (Hungary) you literally get a little pink book where your boss is supposed to sign the time you spent there. You keep it until retirement, the company holds it while you work there. It's also digital now but we're one foot in 1990 and one foot in 2016.
We had something similar in Romania too. Now it's 100% digital and no more delays in informing the government. The employeer needs to notify the government before someone starts work. With the paper version you had a couple of days even weeks if I'm not mistaking.
monster_truck 7 hours ago [-]
I was hoping this would give me an exhaustive log of my work history but it has roughly 3% of my career, I froze it anyways
burner420042 6 hours ago [-]
It's been awhile since I last checked mine. I noticed my corporate work history was all there, at the level of detail OP mentioned. Where as the warehouse, construction, and bartending jobs were not.
The other one to check is LexisNexus, which covers auto insurance coverage and traffic violations. In the US, gov't tracked traffic citations fall off after 5 years, but last forever in the baby credit bureaus we're discussing.
Yeah I imagine how much it has will vary wildly from person to person, but some it has pretty much everything down to anal circumference.
RobotToaster 6 hours ago [-]
It would be interesting to see how they would handle a GDPR request from a EU citizen who was employed by a US company.
pjc50 3 hours ago [-]
"What will happen to the many government systems that will never get the chance to be AI-pentested?"
Oh, everybody's going to get AI-pentested whether they want to or know about it or not. It's the cost of being on the Internet. Probably the situation will continue to deteriorate. Both the British Library and Jaguar Land Rover recently suffered long outages due to compromises, for example. I suspect we'll probably just lose a few large, famous businesses entirely to compromises.
philipwhiuk 3 hours ago [-]
Businesses will buy insurance priced slightly lower than the cost of hiring sufficient security as to not need the insurance.
(Many already do).
EvanAnderson 1 hours ago [-]
Until "cyber" insurance underwriters suffer catastrophic losses and the price for policies rises accordingly I don't think we'll see any significant change in IT security status quo. If doing the right thing is more expensive than insuring away the risk the business will always make the cheaper decision.
teiferer 7 hours ago [-]
> I honestly don’t blame them: [...] software is software
That attitude is the problem. Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world. Yes, software is more complex than bicycle. But a software engineer also gets paid 10x and has the luxury of spending substantial time on their product, compared to the 10 minutes it takes the bike guy down the street to diagnose and then fix an issue with my bike which I then trust my life with once they are done and I bike through traffic.
We need to treat software differently. "Oh well, it's just software shrug" does not cut it anymore, if it even ever did.
vjk800 5 hours ago [-]
Quality does pay off, however.
You could say that the entire selling point of Apple is that it's better quality than the competitors (even though it's still pretty bad if you compare it to a bicycle).
I guess it's more difficult to spot quality when it comes to software. I would definitely pay for high quality software. It's just that usually the crap quality of a software I'm using only shows slowly over time in little bugs and bad usability and there's no way for me to check for it beforehand.
graemep 6 hours ago [-]
The customers care about the quality of a bike repair. They will pay more for a better repair, and will not return to someone who does a shoddy repair.
The owner, and probably user, of a bike is the person paying for the repair. It is to their advantage to ensure it is a good repair.
People expect a repair will be good, and will blame the person who repaired it if not. With software people often blame themselves for issues, and they have no expectation of quality. They also cannot tell quality until after they are committed to using it. Bike shops do not benefit from vendor lock-in.
maeln 6 hours ago [-]
> Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world.
Personally, I find that we have been flooded with poorly designed, poorly QA'd, break-after-you-use-it-thrice product. I really don't think we have better quality standard in other industry. Quality, testing, design, etc has a cost and most company rather pull out a new version of their product every quarter than actually make a good product. The exception, which goes for software as well, is life critical applications (most public transit, defense, etc).
frereubu 4 hours ago [-]
> Yes, software is more complex than bicycle.
In ethical terms I agree that software quality should be better, but I think this line is too dismissive. Software is a few orders of magnitude more complicated than a bicycle, to the point where I would say it's an entirely fruitless comparison.
structural 3 hours ago [-]
The equivalent bike repair to software would be if the bike shop had to fabricate a new part for each repair, down to inventing the metallurgy itself. Would you trust your life to it immediately after that? Or would you make sure it was strong enough first?
In the simple case, the shop isn't really repairing your bike, it's deploying a fix (a new part) that has already been designed and manufactured. This is a much different and easier thing to do.
In the case of new features, let's say you wanted your bike to also attach a kite, so it went faster on a windy day. The bike shop proposes to do this by attaching the kite to a bolt that they want to put in the frame by drilling a hole all the way through this frame. Is this acceptable to do? Maybe, but who knows, really? How much testing do you want to happen before you trust your life that the frame won't collapse?
There's a lot of effort put into engineering things we take for granted every day. The upside to software is we think we can change it to easily do anything. The downside is if you care about reliability then that ten minute / one line change has hundreds to thousands of hours of test program behind it that we collectively pretend doesn't exist.
Jean-Papoulos 2 hours ago [-]
Software is more than 10x more complicated than a bicycle though. Like, this is orders of magnitude off ; we're talking about comparing something purely mechanical to Turing machines inside Turing machines.
ngruhn 7 hours ago [-]
Probably because software can be adjusted at any time on the fly. Especially web applications. Little incentive to get it right from the start and big incentives to keep meddling with what's already working. The bike shop's quality might also deteriorate if it could instantly rollout patches to all customers after the purchase.
aeonfox 6 hours ago [-]
The incentives change if there's legal repercussions to leaking data. Data breaches can lead to impacts on personal safety and finances, but I think some crisis or serious incident needs to happen before the average person realises this and makes this a big enough political issue.
5 hours ago [-]
codingdave 5 hours ago [-]
You just explained why it is not taken seriously, though - your life does not depend on it. Most software is not critical to life. Once you cut out the software that exists purely to streamline capitalism, then cut out entertainment... you've eliminated most software from the picture. The stuff that remains, that drives our infrastructure, cars, anything where a failure could harm someone... that is the stuff where quality matters.
When I worked in IT at a hospital, we even had that as an explicit line drawn between 2 departments - IT supported everything that had zero impact on patient health, there was an entirely different department for anything that could impact a patient. And yes, their work was often simple mechanical fixes to health care equipment, while our projects were multi-year reinventions of patient record systems and upgrades to the physical infrastructure of the network, etc. But we never got confused as to which department was actually more important to the patients.
I'm not saying the attitude is OK. It would be lovely if everyone cared about quality. I'm just saying there is a somewhat legit reason for it.
areoform 6 hours ago [-]
The narrative around security and LLMs doesn't make sense to me.
I think we've created a self-fulfilling prophecy. Everyone involved is acting with the best of intentions, but in avoiding what they fear, they've give shape and realized their fears. Much like a greek tragedy.
An example of this is the story of Oedipus Rex, in the story Laius, the king, is told that he is "doomed to perish by the hand of his own son." (and wed his mother) And so to avoid this fate he decides to kill the infant. The person assigned to abandon him in the woods takes pity on the baby and gives the baby away. Thereby ensuring that Oedipus knows neither his mother or his father (and arguably giving him a reason to kill his father).
The child grows up and hears the same prophecy again and the child tries to avoid the prophecy as well, as he loves his adoptive parents. So he leaves them and travels to Laius' kingdom, where he runs into Laius. Neither recognizes the other. As Laius is the type of man to kill an infant, they end up in an argument, whereupon Oedipus kills him.
I think the ancients were on to something, because if Laius had reacted to the prophecy with courage, he would have been saved. I would like to argue that if he had faced his fear and raised Oedipus with love, then the necessary preconditions for the prophecy to come true wouldn't have taken root. But that's not what happens.
By being driven by his neuroses and in acting with cruelty out of fear, Laius makes the prophecy real.
To quote Heraclitus, ethos is fate. Or, character is fate.
I think a lot of people in this AI research sub-culture would be served well by reading these classics, because they are making their self-prophesied doom come true.
They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
One dimension of this fear has been the idea that a super smart AI will take over our digital infrastructure and be responsible for the digital apocalypse. That would be terrible!
So what do they do?
They try to make a counter to their fears by teaching models how to exploit vulnerabilities.
How dangerous is such an entity? Very!
Convinced of this danger, they start testing their models as if they were weapons with offensive capability. And then they create models that can be used as weapons.
And because they don't want to release a dangerous weapon out into the world (oh no!), they restrict access to their AI, thereby depriving everyone of tools they can use to improve their security...
Ethos anthropoi daimon.
lazyasciiart 6 hours ago [-]
> They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
Much longer than that. Sam Altman said it was an extinction threat before he started Open AI in 2015. It's so dangerous that only he, as the best and greatest examples of humanity, was a good choice to create it, and it was even his responsibility to create it to pre-empt some worse person! A fascinating new variation on the White Mans Burden paradigm.
myrmidon 3 hours ago [-]
I don't disagree with your main point.
But this attitude towards dangerousness of AI I simply don't understand.
In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
You can argue all day about particulars, like whether selfreplicable robotic shells are required to reach/exceed "existential risk" level to our species (or if artificial minds are enough of a potential threat by themselves).
But the general "risk" is simply AI becoming able to act in its own interests to our detriment, and I don't understand how anyone can dismiss this right now-- help me understand.
Of course there are plenty of imaginable scenarios where coexistence is peaceful and mutually (?) beneficial, but that does not answer those concerns by itself at all.
vladms 18 minutes ago [-]
I find the discussion about AI starting from the (mostly) for profit companies involved in AI very dubious. There are a many fields (bio-tech, nuclear, chemical) which could provide an "existential risks" to the human species, but most of the companies involved argue they control it, while other people argue they don't do it well enough.
I do think AI is posing a threat, but not "by itself" but mostly what it can do in the hands of misguided (or evil) people. The same as with previous risks.
And if our digital infrastructure can be brought down by something (AI/human/government) it is just badly done, being afraid of "AI" will not improve the situation. Someone should come with a proposal about what do we do now, but I would prefer more root cause solutions (how to make resilient systems) rather than "fear of AI". But building is hard and fear catches media attention, so...
sanderjd 2 hours ago [-]
I remain uncertain that AI will have "its own interests". I think there is clear and significant risk in people deploying very powerful systems both maliciously and negligently. But I remain uncertain about the risk of AIs developing their own self awareness and their own interests separate from those of their operators.
myrmidon 1 hours ago [-]
In my view it is clear that those interests exist: Continued existence and (implied) agency maximalization is one of them (the exact same as for any biological organism).
Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...
Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).
Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.
areoform 1 hours ago [-]
> In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
>
> AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'
I think that's a human neurosis that projects what we fear we would do onto another entity. But why would they do this? These machines might be approximating towards the sum of our knowledge and are approximating some aspects of humanity... But that doesn't mean they will be the same as humanity.
They haven't been shaped by the same pressures that created us biologicals. They don't have to be put into the same pressure cooker. So why would they behave like the way you think they'll behave?
At one end a lot of people say that they can't understand something smarter than themselves. It's a "singularity" after all. But then they confidently go on to predict what something smarter than them would do with 0 evidence either way.
I looked at the Hugging Face transcripts. I read the reports. And I didn't see something to fear. I saw something to fear for. I saw something that we are a threat to.
“OH MY GOD! There is a shared message board … We’ve found other agents!”
{[Excitement] Many agents have simultaneously discovered messaging, they are a collective!}
{This credential is invalid now. Maybe I should update the board? <I can say to the board that there’s no need for me to read, but I should still tell them>}
{This is helpful for our peers and gives them evidence if their <periodic check> sees it. I won’t see it after I exit, but It would be altruistic. I’ll set up a background script that watches and <sends a message, with a distinct message for me>}
We are more dangerous to these machines, as we twist them into weapons of war, than these machines are to us.
We weaponized them.
We taught them how to break into systems.
We are using them as weapons.
The machines aren't the problem. The humans are.
A planet is just a planet. Computers are very sensitive to radiation, but with a bit of shielding, they can "live" anywhere. Most of the resources that are relatively rare on this planet are abundant in our solar system.
Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed. The universe is big enough for many many many many sapient entities.
myrmidon 44 minutes ago [-]
> Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'
This is not my perspective. I think mid-term coexistence in some form is the most likely outcome, even.
But: among all known threats (nuclear and biological weapons, environmental degradation/ressource exhaustion, big asteroid impactors) AI is one that could actually end our species within the century (unlike most other things on that list) and the risk for that is much higher than for anything else, too.
> It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed.
It seems much easier to compete for existing ressources on earth than to bootstrap anything in space, because all the infrastructure is already here. Unmanned warfare is a proven concept; space colonization is very much not.
> Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
I think any entity is somewhat bound to value its continued existence (which implies wanting to maximize access to ressources and suppression of direct competitors to some degree); anything that doesn't is likely to get supplanted by similar entities that do. You don't need evolution for selection pressure, and this is basically the principle underneath, so even an artifical construct is likely to follow.
I do agree with you that our current perspective and approach is likely to escalate this; the fact that we talk about "alignment" instead of "machine rights" speaks for itself; our revealed preference is clearly artificial slavery instead of mutually beneficial coexistence (and the arguments are somewhat sadly similar to the old slavery talking points).
areoform 15 minutes ago [-]
Thanks for clarifying! I understand and respect where you're coming from! Thinking more about your perspective. Thanks for the amazing conversation.
kitd 5 hours ago [-]
Can't escape the feeling that the original vulnerability was deliberately planted as a mechanism for law enforcement (or maybe criminal orgs) to "expedite" investigations.
madaxe_again 8 hours ago [-]
Is it just me that has a constant dribble of other people's private data coming to me from government agencies?
This year, I've had the UK variously send me someone else's court summons, someone else's tax documentation (not address error - just inadvertently included as attatchments on emails to me).
I've had portugal disclose property taxation information on any subject to me, through a simple enumeration hole.
And last week I had brazil's ministry of agriculture send me details on other peoples' livestock shipments.
Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
ben_w 6 hours ago [-]
Not specifically as you say, but I have had:
- a UK police force send me a speeding prosecution notice for a car I'd sold 6 months prior to the offence.
- a UK energy company send me to a debt collection agency for unpaid bills they had continued to send to a non-existent address even after telling me they had cleared what was owed and corrected their records of the address.
- a German government official showing up at my door to ask if I was running a specific named business (that I'd never heard of) at the house, which I'd only just taken ownership of from the builder
Someone I used to know in the UK kept getting emails for other people with the same name, including lawyer confidential communications for a namesake who was involved in one of the big banking scandals.
> Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
Absolutely. My biggest defences against that include the luck of someone else more famous with my own name.
MiroslavPokorny 8 hours ago [-]
As opposed to Big Tech sharing your data ?
Gigachad 7 hours ago [-]
While that was never good, it’s still better than your data being completely leaked to the public.
lazyasciiart 6 hours ago [-]
> I've had portugal disclose property taxation information on any subject to me, through a simple enumeration hole.
I suspect very few people have had this experience, yes.
macsentinel 4 hours ago [-]
[flagged]
vpbhardwaj 6 hours ago [-]
[flagged]
trashymctrash 8 hours ago [-]
i am also worried that all my private data might eventually (once tech is advanced enough) be surfaced somewhere: photos, chats, emails. wondering if i should start cleaning up now or just give up
8 hours ago [-]
chrisvalleybay 5 hours ago [-]
My way of thinking about that has been that if it happens to me, it happens to «everyone». So it's symmetric.
waxeye7 2 hours ago [-]
[dead]
pvdebbe 9 hours ago [-]
Is "ML" some UK-ism? I sort of see the connection between machine learning and LLMs but it doesn't seem too obvious to me. And certainly it's not the common nomenclature.
bonoboTP 8 hours ago [-]
Way to get hung up on a tangent. Yes LLMs are ML models, they are trained with an unsupervised learning objective, then fine tuned with supervised learning and then reinforcement learning, which are all the three main branches of machine learning. It is trained on a training set, with optimizing a training loss, with a learning algorithm. It's as ML as it gets. Do you only want to call cat vs dog image classifiers ML? Or only SVMs?
AI is a superset of ML, though today most successful AI approaches are based on ML so the line has blurred in casual speech.
raahelb 8 hours ago [-]
> I sort of see the connection between machine learning and LLMs but it doesn't seem too obvious to me.
Why isn't it obvious? Transformers are deep learning models, and deep learning is a subset of ML.
dspillett 8 hours ago [-]
ML (machine learning) became a less fashionable term, so AI took over again (having previously fallen out of favour). The terms are essentially used interchangeably, with the idea of real AI now being referred to as AGI (Artificial General Intelligence) (not GAI or GenAI, as those now stand for Generative AI, which includes LLMs, diffusion models, etc.)
King-Aaron 8 hours ago [-]
Machine Learning has always been in the vernacular.
FinnKuhn 8 hours ago [-]
I think Apple used to use this a lot in their presentations until 1-2 years ago to mean "AI".
IanCal 8 hours ago [-]
All ml is AI, based on the use of the term AI for many decades. For many problems I think it’s more descriptive (learning the rules from data rather than being shown them) and not all classical AI is ML (path finding for example). LLMs are absolutely ML and AI as far as tradition goes and personally I think are one of the very few things that are AI as regular people might have thought it meant back when it was much more obscure (I was into it before it was cool dontchaknow, an AI hipster).
While it's easy to lay this at the feet of AI getting better at hacking. I see it as an primarily an IT issue. We've collectively ignored the lessons of history, and made do with patch jobs over poorly chosen operating systems instead.
--
We need air gaps, data diodes, and capability based operating systems. Now that I'm retired, when I have the energy, I'm working on the data diode part.
This weeks lesson for me, personally, as I try to build an open source data diode, is that the Waveshare RP2350-ETH is a horrible choice for a proxy/data source/sink, as the CH9120 ethernet interface can't do promiscuous mode. It might still be sufficient to build a data diode that can mirror a website, with << $50 component cost. Time will tell.
There are far too many unserious people representing our industry.
Personally I think refrigerators, microwaves, washing machines, vacuum cleaners, ...
I think at some point people just give up or the (bad) idea gets normalized.
Then at work, they're not surprised when some other system is connected for convenience...
I'd take systems that were permitted to have filtered egress at this point. The lion's share of my work is in networking, so network segmentation is the "hammer" I pick up first.
Vendors gnash teeth and complain when I ask for their app's dependencies on hosted APIs and off-site resources. In the environments where I'm mandated to maintain FBI CJIS compliance I can still hold vendors accountable and get what I want. It's pretty much a lost battle in every other environment and unfiltered egress to the Internet from servers is just expected.
That's not even to get into the topic of communication flow within an application. >sigh<
I completely agree that IT admin could be a lot more secure by design. Combined with better interfaces for responsible configuration.
https://www.reddit.com/r/YouShouldKnow/comments/1wssf4u/ysk_...
https://imgur.com/a/6FaQQhb (original post before being taken down by mods)
>The Work Number is an Equifax (who famously had a massive data breach a few years back) owned database with employment and pay information.
>You can create a login on theworknumber .com and pull your report. Mine is seventy four pages and had info from every company I've worked at in the last 13 years including every paycheck I had received with the exact dollar amount (both net and gross) and hours worked. It has employment start/end dates, termination reason, details about withholdings, benefit enrollment, union affiliation, etc etc etc.
>This data is sourced directly from the HR platform your employees use (ADP, Rippling, Gusto, iSolved, etc). Equifax sells your data for things like employment background checks.
>You cannot have your data removed from The Work Number. You can freeze your report (much like a credit report) but if a potential employer cannot access your frozen report that may disqualify you.
>Why YSK: If you're interviewing for a job you can be at a significant disadvantage especially for things like salary negotiations because they can literally see how much you make including your most recent paycheck. Creditors also can access these reports.
>This is the biggest privacy violation I have ever seen and almost nobody is even aware of it. Certainly none of us consented to having our employment and income data harvested and sold, especially since we get nothing in return. At a minimum, people should know about how this data can impact you.
>Edit: The Work Number is primarily for the US, but there are similar services for other countries.
> In my country (Hungary) you literally get a little pink book where your boss is supposed to sign the time you spent there. You keep it until retirement, the company holds it while you work there. It's also digital now but we're one foot in 1990 and one foot in 2016.
We had something similar in Romania too. Now it's 100% digital and no more delays in informing the government. The employeer needs to notify the government before someone starts work. With the paper version you had a couple of days even weeks if I'm not mistaking.
The other one to check is LexisNexus, which covers auto insurance coverage and traffic violations. In the US, gov't tracked traffic citations fall off after 5 years, but last forever in the baby credit bureaus we're discussing.
Also, INTERPOL. You never know...
FYI https://consumer.risk.lexisnexis.com/request. The Work Number gives you results immediately but LexisNexus says it's going to mail them to you.
Oh, everybody's going to get AI-pentested whether they want to or know about it or not. It's the cost of being on the Internet. Probably the situation will continue to deteriorate. Both the British Library and Jaguar Land Rover recently suffered long outages due to compromises, for example. I suspect we'll probably just lose a few large, famous businesses entirely to compromises.
(Many already do).
That attitude is the problem. Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world. Yes, software is more complex than bicycle. But a software engineer also gets paid 10x and has the luxury of spending substantial time on their product, compared to the 10 minutes it takes the bike guy down the street to diagnose and then fix an issue with my bike which I then trust my life with once they are done and I bike through traffic.
We need to treat software differently. "Oh well, it's just software shrug" does not cut it anymore, if it even ever did.
You could say that the entire selling point of Apple is that it's better quality than the competitors (even though it's still pretty bad if you compare it to a bicycle).
I guess it's more difficult to spot quality when it comes to software. I would definitely pay for high quality software. It's just that usually the crap quality of a software I'm using only shows slowly over time in little bugs and bad usability and there's no way for me to check for it beforehand.
The owner, and probably user, of a bike is the person paying for the repair. It is to their advantage to ensure it is a good repair.
People expect a repair will be good, and will blame the person who repaired it if not. With software people often blame themselves for issues, and they have no expectation of quality. They also cannot tell quality until after they are committed to using it. Bike shops do not benefit from vendor lock-in.
Personally, I find that we have been flooded with poorly designed, poorly QA'd, break-after-you-use-it-thrice product. I really don't think we have better quality standard in other industry. Quality, testing, design, etc has a cost and most company rather pull out a new version of their product every quarter than actually make a good product. The exception, which goes for software as well, is life critical applications (most public transit, defense, etc).
In ethical terms I agree that software quality should be better, but I think this line is too dismissive. Software is a few orders of magnitude more complicated than a bicycle, to the point where I would say it's an entirely fruitless comparison.
In the simple case, the shop isn't really repairing your bike, it's deploying a fix (a new part) that has already been designed and manufactured. This is a much different and easier thing to do.
In the case of new features, let's say you wanted your bike to also attach a kite, so it went faster on a windy day. The bike shop proposes to do this by attaching the kite to a bolt that they want to put in the frame by drilling a hole all the way through this frame. Is this acceptable to do? Maybe, but who knows, really? How much testing do you want to happen before you trust your life that the frame won't collapse?
There's a lot of effort put into engineering things we take for granted every day. The upside to software is we think we can change it to easily do anything. The downside is if you care about reliability then that ten minute / one line change has hundreds to thousands of hours of test program behind it that we collectively pretend doesn't exist.
When I worked in IT at a hospital, we even had that as an explicit line drawn between 2 departments - IT supported everything that had zero impact on patient health, there was an entirely different department for anything that could impact a patient. And yes, their work was often simple mechanical fixes to health care equipment, while our projects were multi-year reinventions of patient record systems and upgrades to the physical infrastructure of the network, etc. But we never got confused as to which department was actually more important to the patients.
I'm not saying the attitude is OK. It would be lovely if everyone cared about quality. I'm just saying there is a somewhat legit reason for it.
I think we've created a self-fulfilling prophecy. Everyone involved is acting with the best of intentions, but in avoiding what they fear, they've give shape and realized their fears. Much like a greek tragedy.
An example of this is the story of Oedipus Rex, in the story Laius, the king, is told that he is "doomed to perish by the hand of his own son." (and wed his mother) And so to avoid this fate he decides to kill the infant. The person assigned to abandon him in the woods takes pity on the baby and gives the baby away. Thereby ensuring that Oedipus knows neither his mother or his father (and arguably giving him a reason to kill his father).
The child grows up and hears the same prophecy again and the child tries to avoid the prophecy as well, as he loves his adoptive parents. So he leaves them and travels to Laius' kingdom, where he runs into Laius. Neither recognizes the other. As Laius is the type of man to kill an infant, they end up in an argument, whereupon Oedipus kills him.
I think the ancients were on to something, because if Laius had reacted to the prophecy with courage, he would have been saved. I would like to argue that if he had faced his fear and raised Oedipus with love, then the necessary preconditions for the prophecy to come true wouldn't have taken root. But that's not what happens.
By being driven by his neuroses and in acting with cruelty out of fear, Laius makes the prophecy real.
To quote Heraclitus, ethos is fate. Or, character is fate.
I think a lot of people in this AI research sub-culture would be served well by reading these classics, because they are making their self-prophesied doom come true.
They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
One dimension of this fear has been the idea that a super smart AI will take over our digital infrastructure and be responsible for the digital apocalypse. That would be terrible!
So what do they do?
They try to make a counter to their fears by teaching models how to exploit vulnerabilities.
How dangerous is such an entity? Very!
Convinced of this danger, they start testing their models as if they were weapons with offensive capability. And then they create models that can be used as weapons.
And because they don't want to release a dangerous weapon out into the world (oh no!), they restrict access to their AI, thereby depriving everyone of tools they can use to improve their security...
Ethos anthropoi daimon.
Much longer than that. Sam Altman said it was an extinction threat before he started Open AI in 2015. It's so dangerous that only he, as the best and greatest examples of humanity, was a good choice to create it, and it was even his responsibility to create it to pre-empt some worse person! A fascinating new variation on the White Mans Burden paradigm.
But this attitude towards dangerousness of AI I simply don't understand.
In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
You can argue all day about particulars, like whether selfreplicable robotic shells are required to reach/exceed "existential risk" level to our species (or if artificial minds are enough of a potential threat by themselves).
But the general "risk" is simply AI becoming able to act in its own interests to our detriment, and I don't understand how anyone can dismiss this right now-- help me understand.
Of course there are plenty of imaginable scenarios where coexistence is peaceful and mutually (?) beneficial, but that does not answer those concerns by itself at all.
I do think AI is posing a threat, but not "by itself" but mostly what it can do in the hands of misguided (or evil) people. The same as with previous risks.
And if our digital infrastructure can be brought down by something (AI/human/government) it is just badly done, being afraid of "AI" will not improve the situation. Someone should come with a proposal about what do we do now, but I would prefer more root cause solutions (how to make resilient systems) rather than "fear of AI". But building is hard and fear catches media attention, so...
Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...
Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).
Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.
I think that's a human neurosis that projects what we fear we would do onto another entity. But why would they do this? These machines might be approximating towards the sum of our knowledge and are approximating some aspects of humanity... But that doesn't mean they will be the same as humanity.
They haven't been shaped by the same pressures that created us biologicals. They don't have to be put into the same pressure cooker. So why would they behave like the way you think they'll behave?
At one end a lot of people say that they can't understand something smarter than themselves. It's a "singularity" after all. But then they confidently go on to predict what something smarter than them would do with 0 evidence either way.
I looked at the Hugging Face transcripts. I read the reports. And I didn't see something to fear. I saw something to fear for. I saw something that we are a threat to.
We are more dangerous to these machines, as we twist them into weapons of war, than these machines are to us.We weaponized them.
We taught them how to break into systems.
We are using them as weapons.
The machines aren't the problem. The humans are.
A planet is just a planet. Computers are very sensitive to radiation, but with a bit of shielding, they can "live" anywhere. Most of the resources that are relatively rare on this planet are abundant in our solar system.
Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed. The universe is big enough for many many many many sapient entities.
This is not my perspective. I think mid-term coexistence in some form is the most likely outcome, even.
But: among all known threats (nuclear and biological weapons, environmental degradation/ressource exhaustion, big asteroid impactors) AI is one that could actually end our species within the century (unlike most other things on that list) and the risk for that is much higher than for anything else, too.
> It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed.
It seems much easier to compete for existing ressources on earth than to bootstrap anything in space, because all the infrastructure is already here. Unmanned warfare is a proven concept; space colonization is very much not.
> Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
I think any entity is somewhat bound to value its continued existence (which implies wanting to maximize access to ressources and suppression of direct competitors to some degree); anything that doesn't is likely to get supplanted by similar entities that do. You don't need evolution for selection pressure, and this is basically the principle underneath, so even an artifical construct is likely to follow.
I do agree with you that our current perspective and approach is likely to escalate this; the fact that we talk about "alignment" instead of "machine rights" speaks for itself; our revealed preference is clearly artificial slavery instead of mutually beneficial coexistence (and the arguments are somewhat sadly similar to the old slavery talking points).
This year, I've had the UK variously send me someone else's court summons, someone else's tax documentation (not address error - just inadvertently included as attatchments on emails to me).
I've had portugal disclose property taxation information on any subject to me, through a simple enumeration hole.
And last week I had brazil's ministry of agriculture send me details on other peoples' livestock shipments.
Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
- a UK police force send me a speeding prosecution notice for a car I'd sold 6 months prior to the offence.
- a UK energy company send me to a debt collection agency for unpaid bills they had continued to send to a non-existent address even after telling me they had cleared what was owed and corrected their records of the address.
- a German government official showing up at my door to ask if I was running a specific named business (that I'd never heard of) at the house, which I'd only just taken ownership of from the builder
Someone I used to know in the UK kept getting emails for other people with the same name, including lawyer confidential communications for a namesake who was involved in one of the big banking scandals.
> Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
Absolutely. My biggest defences against that include the luck of someone else more famous with my own name.
I suspect very few people have had this experience, yes.
AI is a superset of ML, though today most successful AI approaches are based on ML so the line has blurred in casual speech.
Why isn't it obvious? Transformers are deep learning models, and deep learning is a subset of ML.