Here's My Type, So Initialize Me Maybe (mem::uninitialized is deprecated)
gankro.github.io/blah/initialize-me-maybe/Posted May 21, 2019, 7:14 PM UTC
Posted May 21, 2019, 7:14 PM UTC
You know, your run of the mill three-state boolean.
I expected this link to go to https://thedailywtf.com/articles/OurBoolean, but having an actual tribool is such a popular library is even more amazing.
I was expecting https://thedailywtf.com/articles/What_Is_Truth_0x3f_
Your link is broken on Reddit's official mobile client, as it doesn't include the trailing underscore. Except it does in the comment editor view. Reddit, why do you have two separate URL parsers in your app‽
Same reason they have two separate Markdown parsers: raisins
At least the result is a Boolean
Reddit actually has two separate Markdown parsers, and the new one is written in Rust. :P
I found this dailywtf article a great way to explain why Option and Result exist and how nicely they compose.
Eh, I actually meant this one and copy-pasted the wrong URL :)
maybe instead of tribool, they could have called it a łukasiewicz
Why stop at 3 when you can have 5?
https://docs.microsoft.com/en-us/dotnet/api/microsoft.office.core.msotristate?view=office-pia
I’m convinced that I learn more from these posts and threads than entire semesters of college.
That's the point. You shouldn't study only at universities, you should to study everywhere.
Ignoring most university courses and only doing enough to get passing grades while learning from reading stuff online and doing a lot of hands-on work in open source has worked out very well for me. I'm way better off than people who studied more. But hey, maybe I just had a bad university.
Paying close attention in most university courses and diligently doing all the exercises while discussing with other students and searching online resources when the lecture did not satisfy my curiosity has served me very well. ;)
But different people learn in different ways.
Do these same arguments apply to mem::transmute()? With transmute, we can just as easily produce:
- dangling/null references
- null fn pointers
- a bool that isn't 0 or 1
- an undefined enum discriminant
- a char outside the ranges [0x0, 0xD7FF] and [0xE000, 0x10FFFF]
- A non-utf8 str
Indeed! And that's why mem::transmute is unsafe. The difference is that it's possible to uphold these invariants using mem::transmute, whereas the value returned by mem::uninitialized breaks the invariants just by existing.
For example, let b: bool = unsafe { std::mem::transmute(2u8) }; is UB but let b: bool = unsafe { std::mem::transmute(1u8) }; is not. On the other hand, let b: bool = std::mem::uninitialized() is always UB, and there's nothing we can do to correct that.
By comparison, MaybeUninit lets the compiler pretend that the value is just () until after you've had a chance to initialize the memory to a valid value.
On the other hand,
let b: bool = std::mem::uninitialized()is always UB, and there's nothing we can do to correct that.
I don't understand why that's always UB. I thought uninitialized was the same as something like this in C/C++, which I doubt is UB.
bool x;
x = true;
What's the difference?
For the llvm perspective, I need to ask a clang expert about this, but my gut reaction is that clang attaches metadata about bools to loads or something, but rustc doesn't because it has a different (in development) model of how memory and type layout works. (tagged union optimizations are a heck of a drug)
edit: yeah it looks like clang lowers bool allocations as unadorned i8s, but when it loads them into a "register" it inserts a truncation to an i1:
%1 = alloca i8, align 1
...
%4 = load i8, i8* %1, align 1
%5 = trunc i8 %4 to i1
https://gcc.godbolt.org/z/15Qw5A
From the pure language perspective, it's a matter of the two languages having different models. But also people thought you could do things like fn return_it() -> bool { mem::uninitialized() } which you can't at all express in a defined way with C++.
Thanks for the explainer and for the 30 minutes I spent fiddling on godbolt with MaybeUninit :)
It's not UB in C/C++, but it is in Rust.
On the other hand, some things are UB in C/C++, but aren't in Rust. Examples include signed integer overflows, integer division by zero and, from the blog post: type punning through unions.
In Rust, the equivalent is safe:
let x;
x = true;
The difference is that, at least for Rust, there's a difference between "uninitialized" and "undefined". "Uninitialized" is tracked by the compiler. "Undefined" isn't.
When you declare and assign a variable in two steps, you aren't promising that the value is well-formed in-between. (In fact, it might not even have space allocated yet, depending on the optimizer.) But when you do let x: bool = mem::uninitialized();, you're filling the x slot and promising that the value in the x slot is always a valid bool. And because we tell LLVM that, and undef isn't, UB results.
I'd like to point out that misuse of uninitialized memory has already caused at least one security vulnerability. It's also notoriously hard to detect you're using it improperly - I literally had to write a custom tool to do that and it's still a far cry from catching everything.
The funny thing is, requesting zeroed memory from the OS is usually exactly as fast as using uninitialized memory. So there is pretty much no reason to use mem::uninitialized, ever.
While the performance equivalency maybe true for rust running in an OS, that isn't guaranteed outside of that. For example uninitialized memory in embedded systems can be faster since you would (on certain systems) have to be the one to zero it out yourself.
However, if you are writing something that is running on linux/mac/windows, you probably should avoid uninitialized memory if for nothing other than security concerns.
requesting zeroed memory from the OS
On Unix, this means using calloc instead of malloc. But mem::uninitialized and MaybeUninit are often used on the stack, where malloc isn’t used in the first place.
alloca could have a similar effect on the stack (non-zeroed memory)
alloca also has a host of issues itself, so you're really trading off one set of problems for another.
The funny thing is, requesting zeroed memory from the OS is usually exactly as fast as using uninitialized memory
Not in my experience. I have some code that got a >100x >10x speedup from switching from doing vec![0; len] and subscripting to reserving and pushing. __memset_sse2_unaligned_erms dominated the profile.
Interesting! which platform and which Rust version was that on? Did __memset_sse2_unaligned_erms run in userland or in the kernel?
Sorry, 100x was a bug on my part when I rewrote it just now. The actual speedup is only about 10x. This is x86_64-unknown-linux-gnu, Rust 1.34.2 (I probably first did it on 1.32 though). It ran in userland, taking >91% of the execution time. With reserve+push, it isn't on the profile at all.
Huh, apparently it was using malloc() + memset() in userland instead of calloc(). What was the type of the vector?
It's known to not lower to calloc() for types that do not implement the IsZero private trait. There is a PR outstanding for some of them.
I wasn't 100% sure if I could claim that
arr[i] = xdoesn't create a reference, so I just used pointer arithmetic to be safe.
If we're worried about that one, should we also be worried about results.as_mut_ptr() implicitly creating an &mut self, since it's an &mut self method? Vaguely related, does the language provide any way for me to get my hands on a *const T or a *mut T directly, without first creating a reference to something?
Unrelated question: Is there any particular reason MaybeUninit's init variant needs to be ManuallyDrop<T>? Or does that just simplify the implementation, since writes to that variant don't have to worry about accidentally running Drop on uninitialized data?
a) It is perfectly fine to create an &mut MaybeUninit::<bool>::uninit() because it is initialized -- to uninit: ().
b) For raw referencing, you want this RFC https://github.com/rust-lang/rfcs/pull/2582
c) To the best of my knowledge the presence of ManuallyDrop is paranoia about well-defined language specification (e.g. they might as well put use it To Be Safe). Although I haven't been following the RFCs for making unions able to contain types with destructors, so it's possible the latest version requires this.
Re: (c), there was a change to unions that just banned types with destructors, to simplify the rules around assigning to the union and/or its fields.
Indeed, with this RFC we banned unions with fields that need dropping. That's why MaybeUninit has a ManuallyDrop.
If we were to start over with the std::io::Read trait tomorrow, would it make sense to instead have this function declaration?
fn read(&mut self, buf: &mut [MaybeUninit<u8>]) -> Result<usize>
Perhaps a unsafe fn read_unsafe(&mut self, buf: *mut [u8]) -> Result<usize>; with better ergonomics surrounding creating raw pointers to uninitialized memory. (see my top-level post below)
It could be implemented in a backwards compatible fashion with a default implementation which zeroes the memory before calling the read function on it.
So the point is to avoid unsafe here without the overhead of initializing memory, and maybe this could enable safe code in both consumer and producer side?
fn read<'b>(&mut self, buf: &'b mut [MaybeUninit<u8>]) -> Result<(&'b mut [u8],&'b mut [MaybeUninit<u8>])>
To quote the Rustonomicon, it is Undefined Behaviour to produce an invalid primitive value
Why? Why did they define it this way, instead of saying that only reading invalid value is UB?
A "read", or "use" is a very intuitive thing that very quickly falls apart when you try to formalize it and people start asking you questions.
let x: bool = mem::uninitialized();
Ok let's say that's not a read. Makes sense.
Is this?
fn foo() -> bool { mem::uninitialized() }
Is this?
let x: Option<bool> = Some(mem::uninitialized());
Is this?
let y = Some(mem::uninitialized::<bool>()).is_some();
Is this?
let x: bool = mem::uninitialized();
let y = &x;
Is this?
let x: bool = mem::uninitialized();
i_promise_not_to_look(&x);
Everywhere you say "no that's not a read" you add difficult constraints to the compiler and memory model for very little benefit.
Sure, but isn't it more likely that the person who wrote the Rustonomicon, whoever they are, is just a huge dingus‽
( for those at home, i am the author of both of these things: https://gankro.github.io/blah/ )
I imagine for various reasons, from optimization to sanity to just being simpler
For example, std::mem::size_of::<Option<bool>>() == std::mem::size_of::<bool>() because, since bool can only be 1 or 0, the compiler can use another value to represent None, like 2.
If references can't be null/dangling, you and the compiler never have to check for that, they're safe by definition. If enums can only have the values you say they can have, you don't need to handle them being something else, even as an error, etc. The types are correct by their very nature, so you only have to worry when making them, and for primitives the compiler controls that outside of stuff like type punning, where you have to.
Also, what consitutes "reading" a value isnt easy to define, theres an issue that describes this, linked below
edit: Rust also tells LLVM that primitives have certain values
In the example of writing bools to an uninitialized array of length 16, can you explain what could go wrong? (Assuming you do assign a bool value to every slot in the array). I've never understood what could happen, specifically, since every byte is overwritten before any are ever read.
Also, in the case of an array or Vec of u8, what risk is there, if any, with using unitialized since the bytes are just bytes, if that makes sense?
It's a matter of language specification, and not some specific implementation issue. It's cleaner/safer to have a "no exceptions" model than to have some complex wart in the specification that Very Smart Programmers can technically use correctly but in practice is too hard to use.
So: what can go wrong? Our specification says this is Undefined Behaviour, because it says so, and that means Everything Can Go Wrong.
To clarify, in terms of specification, rust specification?
Does the language also specify order of operations such that any read of the array (as written in the example) could only occur after it's been written to?
I'm perfectly happy for the answer to be "this tool was prone to misuse and it was prudent to remove it." But I find the notion of what seems to be purely theoretical UB to be unsatisfying. Specifically: on tier 1 targets, if you completely overwrite an uninitialised array prior to any reads, are there any practical dangers? If so, what are they?
I'm not aware of any miscompilations, but what we were feeding down to llvm was seemingly giving it permission to consider the code immediate UB and do whatever to it, as described by the article. Best not to keep that ticking timebomb around.
We did have miscompilations with things like mem::uninitialized::<!>() (this was in generic code, e.g. in rc::Weak). And I think we also had trouble around uninitialized references (conflicting with the non-NULL assumption).
The risk is that before every value is assigned, the bools can be thought to be in a "Schrödinger state" of, for example, 0x02, which is not an allowed value for bools. The point isn't whether it actually is, but the mere possibility is UB.
I am still unhappy with the current design.
It makes working with uninit memory even more verbose than it already was...
You say it's a good thing? No! I agree that 'uninit memory' is bad and should be shunned, however it is sometimes necessary to use as a building block to create safe patterns.
My number one use case for uninit memory are out parameters you encounter in C APIs. One of my uses cases of Rust is talking directly to C APIs without first building a rigorous Rust wrapper API (sue me :P).
Working with out parameters from C API was already precarious and I wish I didn't need to use uninit for this case. But Rust's only support for this is forcing me to use uninit memory and this change makes it even more annoying to do so...
Here's my wishlist for uninit out variables for the purpose of calling C APIs:
// Notice Rust already supports uninit variables!
// This is safe because the compiler checks all access.
let mut x;
// New! Let me create a raw pointer to uninit variables to pass to C APIs:
unsafe { C_API(&mut x); }
// New! Let me tell the compiler that the variable has been initialized:
let x = unsafe { std::mem::initialize(x) };
I would much prefer this over the current verbose dance with MaybeUninit which requires me to write down the types of the uninit variable whereas before with uninitialized that type could be inferred. I see this as a UX regression.
I know that my suggestion is probably much harder to implement as it would require adding special cases to uninit variables.
We should be looking where people are required to use uninit memory and figure out if we can capture these patterns in the Rust language itself where previously not supported instead of just slapping them around with unwieldly verbose syntax and just making it more unwieldly because it has some issues.
Well that turned into a Rant... Idk how to communicate this and last time I had a chance to talk to RalfJung all I got was a somewhat condenscending 'you can't possible use this correctly therefore we must remove this functionality' when my point was, please enable me to call C APIs. I don't mind not using uninit, but please give me something to replace it.
Fortunately I don't get the impression that MaybeUninit is designed to be the end-all solution to every imaginable use case for uninitialized memory. Rather it exists so that the Unsafe Code Guidelines group can formalize any well-defined semantics for working with uninitialized memory, which was impossible with the old API. Once we have a better-understood model of how we want unsafe Rust code to work, I don't see a reason why people couldn't consider additional APIs for making such use cases more ergonomic.
I am sorry that we had a bad interaction. Maybe I can make up for it a bit with this response?
mem::uninitialized()The blog post here explains again why mem::uninitialized() as we had it was a mistake. I appreciate it was very convenient from an API perspective, but it destroyed some fundamental properties of our type system. For types such as bool or &T, nobody has suggested a scheme yet that would be able to "salvage" the old mem::uninitialized() without seriously impeding the compiler's ability to use types for its reasoning. If there are proposals for this, I'd be excited to hear them! We just recently had a brief discussion in the tracking issue delving into some way of saying "a bad bool is only UB when you use it", which unfortunately falls apart when one tries to define "use" properly.
Maybe you are only using this with types consisting of integers and raw pointers. Then there are solutions that would keep mem::uninitialized() well-defined. This is being discussed by the unsafe UCG WG. As usual there is a trade-off here between allowing more code and making it easier to reason about code.
&uninitYou are making a proposal here not to save mem::uninitialized but to provide a more ergonomic alternative. It is a variant of &uninit or &out which has been suggested several times. This is connected to the ancient discussion around "placement new" which goes back to pre-1.0 days. The reason this hasn't happened yet is not that people don't want it. Many people want this! But it turns out to be hard. Here's a blog post giving an overview.
I can only encourage everyone with an interest in this to join the effort here. Getting a type-safe "placement new" for Rust would be truly awesome!
However, given that "placement new" has been in the works for 4 years and mem::uninitialized has repeatedly caused soundness issues and crashes, it does not seem prudent to block deprecating mem::uninitialized on finding a solution for "placement new". We felt we couldn't wait another 2 years, we had to do something. Nobody is happy that this regressed ergonomics for some cases, but this is the best we were able to do.
MaybeUninitHowever, I also have to push back a bit on what you say about the relative ergonomics of MaybeUninit for "out" pointers (though not all of this might apply to your particular cases). You say that with MaybeUninit, one has to explicitly give a type. That is not correct:
use std::mem::MaybeUninit;
unsafe fn mk_vec(ptr: *mut Vec<i32>) {
ptr.write(vec![1,2]);
}
fn main() {
let mut v = MaybeUninit::uninit();
unsafe { mk_vec(v.as_mut_ptr()); }
let v = unsafe { v.assume_init() };
}
main consists of three lines corresponding exactly to the three lines in your wishlist. Admittedly, the first line is more verbose, but the other two are not.
Also, while this scheme takes one line more than mem::uninitialized, it is also more correct! With mem::uninitialized, if mk_vec panics before initializing ptr, v would drop an uninitialized Vec. To do "out"-pointers for types that need dropping requires the use of ManuallyDrop, mem::uninitialized was not enough. Compared to that, MaybeUninit actually wins in terms of number of keystrokes. Or you could prove that your code does not panic, but that's an extra footgun that will inevitably go off at some point. (In terms of teaching, I'd rather teach one scheme that always works than two where one requires very careful control of the environment.)
So, from what I can see, we have an ergonomics regression "only" in the case of types that don't need dropping and where all fields are integers or raw pointers. For all the other cases, the more ergonomic prior approach was incorrect and has lead to miscompilations already -- miscompilations that with the old approach we don't know how to fix without impeding optimizations of safe code everywhere. For the mentioned cases, the regression consists in one added line, the let v = unsafe { v.assume_init() }; in my example above. For this, you gain the aid of the type-checker to make sure you are not mixing up initialized and uninitialized values.
Hey Ralfj! Thanks for taking your time to reply.
I tried breaking up the wall of text in topics to hopefully make it easier to read.
Is the following statement correct? Rust wants to abstract LLVM but because of LLVM's presence some of its semantics leaks through to Rust whether you want it or not. Specifically UB is very hard to contain (see other instances such as 256.0f32 as u8 being UB). In this way you are trying to create a model for Rust that both wants to stand independent of LLVM's semantics but at the same time recognize the value of LLVM (and UB) when it comes to optimization.
The especially aggressive approach of LLVM to exploit UB leads to a lot of frustration (even more common in C/C++!) as well as on your side to try to make all of this work :P
Whenever I'm dealing with reading eg. binary file formats with booleans in them I've tend to interpret them as '0 = false, anything else = true' (instead of eg. panicking/exception/UB). This is in part because I know this lowers very efficiently to the underlying hardware, tests against zero/non-zero are basically free. From this perspective the value proposition of boolean as 'false/true/undef' is a lot less straight forward.
All that said, invalid boolean values have always been UB (even in C++ where I come from) so this isn't that big of a deal since I'm already used to just 'deal with it'.
Yup, generalizing uninit/out references is hard. Even in languages that have it, C# in my case, the compiler is very quick to call you out: 'hey you must initialize an out variable before returning'. Which I promptly broke by throwing an exception just to see what would happen. The out value turned out to be zero-initialised (as all values are in C#) but clearly this solution isn't satisfying.
Trying to fix it by requiring that the code must not panic in Rust doesn't work as Rust has no way to identify and enforce non-panicking code, especially not in a zero-cost manner.
It is true then that I don't even try to fix the general case and suggest an approach which I've only thought about in the use case of calling C APIs. Obviously too much special casing is not a good idea for a general purpose language, everyone likes cross-cutting features that interact well with the rest of the language.
Hmm you're right that type inference seems to work perfectly on MaybeUninit!
I... can't quite remember why I thought this was going to be a problem. I seem to remember it had something to do with raw pointers and arbitrary self types and type inference being dumb:
let ptr = 0 as *const _;
if ptr.is_null() {}
let ptr: *const u8 = ptr;
I've always had a similar issue with closure parameter types:
let f = |i| i.abs();
println!("Hello, {}!", f(3.1415f32));
So uhm my biggest complaint is then resolved, I think.
When dealing with pure Rust code I've come to enjoy &mut Option<T> to stand in for out parameters. If I'm worried about having to drop the original value when I know it is None you can use the replace method and forgetting the old value:
let mut output = None;
mem::forget(output.replace(Some(value)));
I encounter this pattern with APIs that take an FnMut closure as argument but do not have a place to return a value:
fn action<F: FnMut()>(mut f: F) {
f(); // Do stuff
}
let mut result = None;
action(|| {
mem::forget(result.replace(vec![42]));
});
let result = result.unwrap();
In any case this solves panic safety by the Option representing a runtime 'flag' whether the variable is initialized.
My point is that Option<T> seems to have some similarity with MaybeUninit<T> where the former is runtime checked and the latter is the unchecked variant of this use case. I enjoy this kind of relation because it allows to view the same concept in different ways which often leads to better understanding.
Have you considered naming the type Uninitialized (or even just Uninit)? Conceptually this type does not represent an initialized value at any time because the wrapped T is further wrapped in ManuallyDrop. It does not behave like a 'maybe initialized' unlike eg. Option.
This isn't really a complaint but more about feelings. For what it's worth I feel this more closely aligns its name with its mem::uninitialized<T> heritage.
My observation is that the mem::uninitialized code can be transformed fairly mechanically to use MaybeUninit which leads me to think, why should I be in charge of doing this mechanical transformation? Can the compiler do it for me somehow?
The 'wishlist' syntax actually matches MaybeUninit pretty dang closely:
let mut x = mem::MaybeUninint::uninit();
C_API(x.as_mut_ptr());
let x = x.assume_init();
vs.
let mut x;
C_API(&mut x);
let x = mem::initialize(x);
The remaining difference in semantics is that in the MaybeUninit case x is a wrapper type whereas in my wishlist example it is the type itself. I've always found wrapper types in Rust less ergonomic than I'd like especially once you start having arrays of these types or passing them around function pointers with these types in their signature.
Comparing to the hypothetical &out, &uninit references or placement new: panic safety is resolved by not invoking Drop at all even when you've assigned a value until you decide the value is properly initialized (3rd line in both examples). Assigning to an out reference does not drop the value behind the reference.
Why not also compare it to using an option:
let mut x = None;
rust_API(&mut x);
let x = x.unwrap();
The point I'm trying to make is by comparing these solutions with each other we can see how one method resolves a particular question about semantics to better understand the trade-offs in other methods.
I'm happy to understand that MaybeUninit isn't as bad as I thought.
The mechanical nature of the transformation gives me hope that this can inform the debate about how out parameters can be implemented in the language itself and perhaps lead to better ergonomics surrounding out parameters in general and C APIs in particular.
Thanks for taking your time to read!
Hi! I agree with most of what you said so this response should hopefully not become too long. ;)
boolIs the following statement correct? Rust wants to abstract LLVM but because of LLVM's presence some of its semantics leaks through to Rust whether you want it or not.
I would disagree. Rust wants to have an abstract semantics, but it doesn't just want to abstract LLVM specifically -- we want to support other backends as well.
For bool specifically, there are many reasonable choices for what the representation could be. Saying that "not 0 means true", as you suggested, is one of them -- it solves some problems but introduces others. For example, now two booleans that are equal (both true) could actually have different representations (so memcmp would say "not equal"). Comparing booleans requires normalizing them first.
Also, bool was just the example in this discussion. The same issue with "there are some inherently invalid bit patterns and we want the compiler to exploit that" applies to references (cannot be NULL), and enums (must have a valid discriminant). Using types to reason about which shapes your value can possibly have is inherently useful for program analysis, optimization but also verification alike.
&mut Option<T>When dealing with pure Rust code I've come to enjoy &mut Option<T> to stand in for out parameters. If I'm worried about having to drop the original value when I know it is None you can use the replace method and forgetting the old value
If you are using Option anyway (and thus have some run-time checks), wouldn't it be better to panic if there is a Some as opposed to leaking it?
let mut output = None;
let old = output.replace(Some(value));
assert!(old.is_none());
A helper method would probably help. (I've sometimes wanted to "assert is_none", similar to how unwrap does "assert is_some", when inserting into a HashMap and also found that annoying to write.)
My point is that Option<T> seems to have some similarity with MaybeUninit<T> where the former is runtime checked and the latter is the unchecked variant of this use case. I enjoy this kind of relation because it allows to view the same concept in different ways which often leads to better understanding.
Yes, there definitely is a relation. Though MaybeUninit is in some sense more powerful since it can be "partially initialized" -- it's not an "all or nothing" thing.
There was an FCP for the name specifically in this PR.
Conceptually this type does not represent an initialized value at any time because the wrapped T is further wrapped in ManuallyDrop.
I don't understand this point. The use of ManuallyDrop inside MaybeUninit is an implementation detail.
Hey RalfJ, sorry about the late reply...
In my particular case booleans are problematic as they pop up in some 'FFI' cases such as reading memory from another process or interpreting files by casting their memmapped contents to a structure. In such cases it is technically illegal to use bool as I cannot guarantee that these bools do not have invalid contents. I've already accepted my fate here and use u8 instead of bool in these cases. (the same issue is present in C/C++ code but some people like myself in the past have generally ignored this and just used bools because it's more convenient)
When dealing with certain 'should never happen' cases I sometimes simply prefer to silently ignore errors. These aren't always big errors but something stupid like a configuration was specified twice. Here's an example. It uses a helper function to walk a configuration tree and invokes a callback when it matches the name. I wrap it to return the value of the configuration node. What happens if two nodes have the same name? Idc, don't do that but it isn't catastrophic.
You could argue that this is an error and should panic but sigh, idk. Panics are very heavy handed and in this case I decided not to panic. Should the original value be mem::forgotten? Here I didn't, but depending on my mood I may do it. It's not a memory safety issue and correctness outside of that is not always the most important thing.
I would argue that the symmetry between MaybeUninit, Option and hypothetical &uninit references is extremely important! In a sense MaybeUninit and Option implement a kind of uninit references but with different choices in semantics in the edge cases:
MaybeUninit implements it by going through raw pointers and unsafe code to let the developer decide when something has been initialized. The semantics when a panic occurs is then defined to forget already initialized MaybeUninit but not yet assume_inited. Thus the 'manually drop' is explicitly chosen as the fallback case when a panic occurs! This isn't an 'implementation detail' but an explicit choice in semantics.
Option implements it by doing a runtime check, if an they're used to emulate uninit references then the value is dropped correctly when the code panics but before the developer has asserted that the out value has been initialized.
That's what I meant when I said that MaybeUninit does not represent an initialized value. I was trying to figure out how out parameters could be built into the Rust language and what their semantics could be. Both MaybeUninit and Option can be used to emulate out parameters, each with different choices for what happens if the code panics. MaybeUninit forgets the value, Option drops it.
It's hard to know if your proposal is workable without it having a more complete description of what the semantics would be. i.e. How does the compiler track the initialization status of memory if the mem::initialize() call is not in the same function as the uninitialized use?
How I describe it here, it would only be usable locally in a function but not in method arguments, yes.
Rust really wants you to use Option/Result and straight returning your result instead of initializing a variable by passing a reference. Which is fine but this specific case doesn't work when the C API dictates the method signature.
Thus the above proposed only does the bare needed things to get this particular use case to work, the point being that once you call the C API you immediately wrap things up how Rust wants you to do things.
There have been proposals to extend the notion of out parameters in Rust itself through &out var references that are references to logically uninit memory. However the story surrounding panic safety and requiring things to be initialized were never quite fully worked out. (reminds me of the trickery you can do in C# with the out keyword).
There are other questions like 'what should the signature of initialize look like?' (probably fn initialize<T>(T) -> T with special compiler magic to make it work with uninitialized variable). If you pass it something already initialized it acts as the identity function?
Another question is what is the type of let mut uninit; &mut uninit what is the type of the &mut operator? Perhaps *mut T. What about the & operator? Probably not allowed. All of this is a ton of special casing just for calling C APIs. Is it worth it? For me, of course yes :P but is it more generally useful?
Perhaps in the definition of the io::Read trait where it is more naturally to express the more general idea of out parameters because you need to provide a buffer of bytes to read into. The idea here would still require unsafe where I think it would be preferable to have such an API not require unsafe code.
All I can really tell is that sure, mem::uninitialized is problematic, but mem::MaybeUninit doesn't solve all use cases and makes the rest less ergonomic.
Say you wanted to apply the simplification y & x => y. That requires proving that x is all 1's. Oh it's uninitialized memory? Ok let's just assume it's all 1's. For y | x you can assume it's all 0's. Whatever's most convenient at the time! More controversially, one may also conclude normally impossible things, like x == x => false, by assuming the value is changing on each read.
If the programmer creates an uninitialized value and does not write to it, then these compiler assumptions don't strike me as ridiculous. You write that:
No, it doesn't matter that we didn't read the uninitialized memory.
I don't entirely follow why this doesn't matter. If it's at all possible that a write to that uninitialized memory occurs along the control flow graph before a read, then it seems to me like it's a compiler bug to assume that the memory is still uninitialized at the time of the read?
If it's at all possible that a write to that uninitialized memory occurs along the control flow graph before a read, then it seems to me like it's a compiler bug to assume that the memory is still uninitialized at the time of the read?
Just the presence of undefined behavior means the compiler can do whatever it wants. Usually "what it wants" is to make the code fast, and it accomplishes that by assuming that the circumstances that would cause undefined behavior don't happen. For example:
use std::hint::unreachable_unchecked;
pub fn test(value: u8) -> u8 {
if value == 0 {
unsafe { unreachable_unchecked(); }
}
100 / value
}
unreachable_unchecked is a compiler built-in that says that it is undefined behavior to execute it. If you look at the generated assembly, it doesn't check at all whether or not value is 0, it just does the division. That's because, if value == 0, there is undefined behavior, so the compiler can act as if it's impossible for value to be 0. If you comment out the unreachable_unchecked, then it's no longer undefined behavior to pass 0 so the compiler generates code that checks for the attempted divide by zero and panics.
Essentially we are passing down to llvm the following lines (pseudo llvm ir because I don't want to look up the proper terms):
var = allocate_local_var()
*var = undef
assume(*var < 2)
// ... later we actually initialize it
*var = 1
The compiler can look at the first three lines and instantly derive a contradiction:
Wait, what? MaybeUninit has been stabilized!?
I dislike how implementation details of compilers, mostly coming from the C and C++ world, are leaking into more safe programming languages API. From the programmer standpoint, there is absolutely no reason for the compiler to try to "destroy" their program by using absurd inferences on e.g. unobserved bools being invalid. In that regard, letting union doing type punning and not having the concept of active union members was the right choice. Arguably, invalid representations of bools yielding to UB is even a poor choice in the general case (I doubt non-trivial real world gains can be achieved thanks to that); likewise for uninitialized memory being considered so much tainted: modeling the real hardware and low level allocators implementations by considering the representation unknown but stable, would obviously yield to less risks, and I'd also be surprised if it currently being UB yields to that much interesting optimizations for valid code.
I would have preferred to see a movement toward making mem::uninitialized well-defined and sound, rather than simply working around the issues (with the drawbacks explained, like it disabling other optims that can have an impact on memory consumption).
Still, I understand that I'm nitpicking and I will happily use MaybeUnit if I have to, but I consider it to be a compromise, and not the Right Choice™.
Thanks for the feedback! I happen to be a big fan of encoding invariants in types, which is why I supported and pushed this approach, but I guess I cannot convince everyone. :)
To reply to some of your concrete points:
Arguably, invalid representations of bools yielding to UB is even a poor choice in the general case (I doubt non-trivial real world gains can be achieved thanks to that)
Without that assumption, every if has to be compiled to a three-way test to panic/abort if the bool if neither true nor false. A jump table or two conditional jumps instead of one seem like some serious extra cost to me.
likewise for uninitialized memory being considered so much tainted
LLVM exploits undef heavily during register assignment, that's (one of the) reason(s) why "uninitialized" values can "wobble" in practice. Not doing this would thus increase register pressure, which is a precious resource.
Unfortunately I am not aware of a systematic study of what these kinds of decisions actually gain in terms of performance. However, I have seen the struggle my coauthors had when trying to get the slowdown in LLVM for our paper down into the low-single-digit percent range.
[removed]
If you read from uninitialized memory, you are essentially reading random bits, and so your program may behave randomly. From this perspective, it's almost always a serious bug to read from uninitialized memory. Although you could certainly construct cases where that's not the case.
...
So as a conservative model it's reasonable to just declare that if you do anything with uninitialized memory other than just copying it around, it is Undefined Behaviour. Full stop.
The first paragraph seems incorrect. My understanding is that it is straight up undefined behavior. It's not only undefined behavior from a compiler perspective, or the perspective of C, but isn't it also undefined behavior from the perspective of Linux or any system that is possibly executing your binary? For example, Linux could, and does sometimes, cause a seg fault?
EDIT: I think I was wrong about Linux possibly causing a segfault. What I was thinking of was that Linux DOES keep track of which pages of memory have been written to. It's possible that reading uninitialized memory multiple times might read different values. Linux feels free to do this because no program should be reading uninitialized memory.
For example:
A seg fault is caused by accessing memory you don't have access to (either because it belongs to the kernel, or hasn't been mapped to the current process), or by accessing that memory in the wrong way (e.g. writing to a read-only address). Apart from that, though, the kernel doesn't really care about what you do with the memory that it's given you.
The examples of mem::uninitialized given in the article are all using values on the stack. This is memory that is allocated, and generally safe to access. In many programs, it will also be memory that has been written to before. Accessing it won't cause a seg fault. If we could get the compiler to turn a blind eye for a few moments, we could use ptr::write to initialize the memory, after which everything would be fine. The problem is that before this point, due to compiler semantics, the compiler is allowed to make pessimal assumptions about whatever that memory contains. And those pessimal assumptions might include values that, according to Rust's assumptions, constitute undefined behaviour.
Undefined behaviour, in this context, refers to situations that the compiler are allowed to assume will never happen. For instance, it's allowed to assume that bool will only ever be 0 or 1, so it can perform optimizations based on that fact. If somehow you manage to sneak a bool into the system that isn't 0 or 1, things might break unexpectedly. A mem::uninitialized bool is 0, and 1, and every other value that could make the compiler's job easier, and the easiest job for the compiler is to assume that doesn't have to actually compile this code.
The compiler is allowed to assume that any code that breaks these rules is invalid and will never be executed, so can be optimized away. Obviously, code following the UB is never going to be run either, so can also be optimized out of existence! Less obviously, this works in both directions: if the UB code is preceded by some non-UB code, that non-UB code won't happen either. If there's an if statement, with one branch leading to UB code, the compiler will assume that branch will never be chosen and will optimize away your if statement. In theory, if all branches lead to UB, your program could be optimized away entirely!
The first paragraph seems incorrect. My understanding is that it is straight up undefined behavior
This section is from a "low level implementation perspective". There's no undefined behavior, there is no compiler to define what behavior is allowed :)
Ultimately you can always write assembly to read from uninitialized memory. That will fine, and it will work, it just won't do anything useful. Which is what this paragraph is describing.
You're right that there's a bunch of randomness that can happen from the kernel here. That's not the same kind of "undefined behavior", in the context of compilers "undefined behavior" is not just for scenarios where the behavior isn't 100% defined, it's for scenarios where the compiler has full license to do whatever it wants, including replacing your stack frame with poo emoji.
Here the OS does not have full license to notice your weird memory reads and replace your stack frame with poo emoji. This falls into implementation defined behavior: There are specific constraints on what the behavior may be, but different implementations may have different choices. In this case, the constraints are that the OS may return whatever it wants on such memory reads (and it doesn't have to be consistent across reads).
I don't think OSes really have the same kind of concept of UB, it's almost always implementation defined behavior.
The semantics of uninitialized memory inside compilers are *extremely* subtle, but regardless, the first paragraph is simply describing why uninitialized memory is a thing from an implementation perspective. Say, if you were hand-writing assembly.
The system, in this case the CPU, has no concepts of types when running the generated binaries, everything is a blob of data that is accessed by instructions.
At most what could be faulty is to read from a random address or accessing some data with an instruction relying on a specific alignment not upheld from the address, but as far as the processor is concerned, there's no bool or Enums that must always hold some specific memory representation.
Undefined behavior means that the compiler can do whatever it wants. The actual machine code that it generates chooses a certain option, whether it's removing the entire code path, segfaulting, printing a warning, or emailing a rude message to the NSA. The option the compiler chooses isn't necessarily the same on any given platform.
Reading or writing memory outside of your assigned address space is what causes a segfault. That means that reading uninitialized memory that you just allocated won't cause one, and neither will overflowing an array on the stack. But writing code that reads/writes out of bounds won't necessarily cause a segfault, since the compiler doesn't have to generate machine code that accesses that memory in cases where it would be undefined behavior.
For example, Linux could, and does sometimes, cause a seg fault?
No. Linux does not know or care about the application-level concept of memory initialization.
Ok, I refreshed my memory. My understanding is that Linux does keep track of which pages of memory have been written to, and something like this can happen:
As far as I can tell, I was wrong about it possibly causing a seg fault, but this seems like a pretty clear indication that as far as Linux is concerned, it can do whatever it wants when someone reads uninitialized memory.
Linux has decided that you're reading uninitialized memory.
Linux does not decide this. Anonymous pages are explicitly zero-initialized. They may be implemented via a ZFOD scheme, but that's transparent to the application. The physical page that Linux allocates will be filled with zeroes before it is mapped into the process's address space.
Linux does not decide to cause a segfault when your program does the wrong thing. Your compiled code runs on the CPU, and if it attempts to break the CPU's rules on memory, it will trigger a CPU exception. This causes it to stop executing your code and jump into Linux's exception handler code, basically just telling Linux "this piece of code just attempted to break the segmentation rules, what do you want to do?" Linux just reacts to the exception by sending a signal to the process killing it, then moving on.
Ideally, you will also read or drop_in_place any initialized memory whose type has a Drop impl when you're done with it, although forgetting to is technically allowed.
What's the reason behind this? I'm sure it's not too complex, but I have pretty much no experience with this.
[deleted]
core::mem::MaybeUninitis stable from 1.36.0 onwards, and core::mem::uninitialized is deprecated from 1.38.0 onwards.
If you still need to support an older version of Rust than 1.36.0, how should you wrangle things? select-rustc could help you support both manually, but it depends on a procedural macro, so I think that’d only gain you support back to 1.30.0 and the 2018 edition. If you need older, I suppose it’d take a build script of some form.
This sounds like something that could beneficially be abstracted into a crate. Use core::mem::MaybeUninit from 1.36.0, an in-crate copy of it from libcore for Rust 1.19.0 (the first version with union) until 1.36.0, and an API-compatible wrapper around core::mem::uninitialized for versions still older than that.
This article is really entertaining.
The Rust Language developers are so clever. They keep coming up with things that make so much goddamn sense. Wow. This is awesome.
[deleted]
that's very clearly sarcastic hyperbole for comedic effect
Also IMO it is legitimately more instructive to regard the compiler as adversarially trying to find UB, as it disillusions oneself from the notion that "well even though there's UB, no compiler would ever miscompile this".
e.g. assuming `x = *ptr` will reliably lower to a cpu load instruction, and therefore result in a hardware trap if `ptr` is null. "Why would a compiler ever break that?" Because it's a complex system designed to find UB and break your program if it finds it.
Ah, yes, this is a good point.
Alternatively, instead of thinking of the compiler as your enemy, think of him as an extremely misguided friend.
"But I just want to make your code go fast! Look, I found another branch that has UB on one side, so I can delete it and make it go faster."
[deleted]
[deleted]
I’d say they go out of their way to prove that something is UB for some possible inputs, which then allows them to simplify the code by assuming those inputs won’t occur. That’s roughly equivalent to what you said, but I think it still counts as “going out of their way to prove UB”.
Although there are definitely some optimizations that should trigger a warning, like detecting and collapsing infinite loops.
But again it's not like the compiler actively tries to prove "this loop is infinite" so that it can remove it. It's rather that it assumes the loop is finite and then bad things happen if it is not.
See this document for why this particular clause is in the C standard, and the optimizations it enables. It is entirely unclear how to turn that into a warning.
Compilers don't emit warning when exploiting UB not because they hate you, but because they don't even know that they are exploiting UB -- they just assume you did your job and avoided UB. Beyond toy examples, it's really hard for the compiler to tell when it incorrectly assumes UB-freedom.
In the code I posted, it turned a loop I wrote with a conditional into a hard-coded infinite loop, then deleted all of the code inside it and after it. It may not technically be accurate to say that the compiler is "proving" anything, but I would certainly hope someone has proven something about that optimization.
Granted, this is a toy example. But it's a relatively common mistake, and it's fascinating to me that GCC has the ability to detect it, but they decided that code was best put to use optimizing infinite loops, rather than putting it in, say, a linter. Because golly, if there's anything worse than an infinite loop, it's a slow infinite loop!
Yes. The only thing is: that's a very bad assumption. We have several decades of practice and all the experts telling us that. Actually Rust would not even need to exist with that kind of assumption.
So yeah, this is helpful for compiler writers to merely assume things instead of proving them. This is not for their users (maybe this is for a few video game programmers who do not have to care much about the risks, but who want the absolutely fastest possible binary, even if it yields to more actual bugs -- but for the general purpose programmer this is not helpful).