Having trouble with a lifetime for a reference passed with the value itself

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=ef95c84f4358107e1bee95f1cae49f57

I solved my previous problems with implementing an enum that supports polling different types of async results, but now I'm running into a lifetime problem.

My thinking is as follows -

(L78) I hold on to the enum values in the remaining futures vec.

(L84-85) We get the values back from select_all in .2 minus the one that completed, and move that back into our remaining_futures variable. next is the value of the future that completed.

(L90) we match and move the value of a ReadDir to rd.

Calling next_entry will be a reference to rd, so we need to ensure that the ReadDir value is kept alive alongside the future, which we do in another field of the tuple.

In my head at least, I'm being careful to ensure that the value is kept alive to be referenced, but I'm really struggling to express that to the compiler (or I'm not understanding something).

Any help very welcome to help me understand what I'm missing here.

Thanks!

Posted Jun 5, 2021, 7:01 PM UTC

67
soruh•

Your problem is that you are attempting to construct a self referential struct which you can't safely do in Rust. You're creating a reference to rd by calling rd.next_entry() and are then attempting to move rd. If the compiler let you do this, the reference you created would become invalid. Also if you were able to construct that struct, moving it would invalidate the reference to rd again.

basically:

Pass rd value around to keep it alive so that the reference for next_entry is valid.

This doesn't work that easily. You could pin rd to the heap (Box::pin(rd)) and then use raw pointers to it in your future. This would require unsafe and thus quite some safety reasoning of course.

This design seems somewhat overcomplicated. Is there a reason you can't just have a tokio task per directory which iterates over it's children and in turn spawns a new task for iterating over each child folders children?

3
Redundancy_OP•

Thanks, that's useful to think about.

I've done a few toy projects in Rust, but I've often avoided some areas intentionally (particularly the borrows and lifetimes). I'm trying to put myself in a position where I've got some non-trivial problem to solve that forces me to work through and understand some things that you just don't hit in rustlings exercises. If I understand the problem that makes the direction of the solution untenable, I've learnt something and can rewrite it in another way until I've gotten through it all, but I'm not just immediately picking the easier paths because I don't have a full toolbox of approaches to solving problems yet.

2
Redundancy_OP•

More out of curiosity than anything, this is possible with metadata, but it seems to come down to the issue that metadata and read_dir use to_owned() to grab an owned copy of the paths. (some code modifications left out, but the main gist...)

        FsFutureOutput::ReadDir(Ok(mut rd)) => {
            println!("{:?}", rd);
            while let Ok(Some(x)) = rd.next_entry().await {
                remaining_futures.push(
                    FsFutures::Metadata(
                            x.path().as_path().to_owned(), // ugh?
                            Box::pin(tokio::fs::metadata(x.path()))
                    )
                );
            }
        }
        FsFutureOutput::Metadata(path, Ok(metadata)) => {
            if metadata.is_dir() {
                remaining_futures.push(
                    FsFutures::ReadDir(Box::pin(tokio::fs::read_dir(path)))
                )
            }
        }
1
SkiFire13•

An alternative could be moving rd inside the future and having it returned when the future completes. https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=f443c0930e104eb2ec61b9b97d7d2251 In particular note the lines 35-38

DirNextEntry(Pin<Box<dyn Future<Output = (
    tokio::fs::ReadDir,
    tokio::io::Result<Option<tokio::fs::DirEntry>>
)> + 'a>>,),

and the lines 99-102

FsFutures::DirNextEntry(Box::pin(async move { 
    let entry = rd.next_entry().await;
    (rd, entry)
}))

Note that technically this is still self-referential, however now it's the job of the compiler to handle that for us.

2
Redundancy_OP•

Oh! I was trying a few things like that but didn't know how to make it work. More stuff to play around with, thanks!

1
Redundancy_OP•

Hey, would you look at that!

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=88afd1a254b6f003777e41bac6a21386

So the answer is that you can, but maybe you shouldn't :D

Thanks, it's been fun and interesting debugging and working on this.

1
Ytrog•

I am on my phone without access to a computer, however I hope someone can help you with it. 😊👍

-1