If you're using sync (local network or dropbox), all newly imported files now have file names that are stable and unique (but not human-readable) names anymore. This means that if you use Papers to sync your library between devices, you no longer have an option to organize your files by, for example, author, title, year etc. This is because we can't technically guarantee where the files referenced in the Papers library are at any point in time. In order to be able to offer reliable syncing of PDFs across different devices we have to disable custom file renaming when the library is stored on Dropbox and have Papers control file naming.
Because it is important to have access to human-readable file names as well, we have developed a "browse in finder" (aka "virtual disk") feature so that we're still able to provide human-readable file names even when sync is enabled. In addition you can also browse your library's PDFs by any of the collections, keywords or labels that you have created.
The "browse in finder" feature is available by clicking the "Browse" button in the "Library" tab of the Papers preferences. When you click the button for the first time, you're asked to install a system component that is required to make the feature work.
Also note that the virtual file system needs Citations to be running in your menu bar. You can also access your Papers library in the Citations menu by choosing the "Browse Library in Finder" menu item.
Please note that the "browse in finder" feature is still at its beginnings and we have many plans for improvement (e.g., Spotlight support is high on our TODO list).
Also note that, if sync is not enabled, you currently still have the same renaming options available as in Papers 2. In this case, the renaming options in the Library tab of the Papers preferences still control the file & folder structure of your library. They also control how the "All Papers" section of the virtual disk is organized.
If you have trouble accessing the virtual file system, try toggling Citations off and back on again by going to Preferences > Citations.