v2.1.0 Content Directory support

What's Changed

This release adds new functionality to the analyze command, buiding on top of the AssetBundle schema cleanup that landed in v2.0.0.

The main focus is introducing support for the new Unity 6.6 "content directory" build system. To analyze these builds properly the ContentLayout.json file should be read at the same time. For convenience a --build-history argument can be used to help match the correct layout file with the content directory build that is being analyzed. The ContentLayout.json is converted into equivalent database tables, and also used to resolve references in the actual built content.

Also, to better "expose" missing content or incomplete information, there is now a "dangling_refs" table. It is expected that references to the "unity default resources" are reported here.

Error handling to analyze is improved, instead of obscure or hidden SQL errors there will be a clear error if more than one serialized file with the same name is encounted.

There are also significantly more documentation explaining the content inside Player and Content Directory builds, and improvements to the AssetBundle documentation.

PRs

For details see the individual PRs that landed:

• Document AssetBundle dependency references by @Abhinav-0311 in #91
• [#92] Add Player build and content directory format documentation by @SkowronskiAndrew in #95
• [#85] Track dangling references in analyze by @SkowronskiAndrew in #96
• [#97] Add Unity scenes to LeadingEdge reference builds by @SkowronskiAndrew in #98
• [#99] Import ContentLayout.json into the analyze database by @SkowronskiAndrew in #101
• [#99] Resolve ContentDirectory references via ContentLayout.json and validate analyze inputs by @SkowronskiAndrew in #102
• [#99] Create build report schema on demand; view rename and Content Directory docs by @SkowronskiAndrew in #103
• [#51] Clear error for duplicate SerializedFile/archive names by @SkowronskiAndrew in #104
• Match SerializedFile names case-sensitively by @SkowronskiAndrew in #106
• [#99] Resolve ContentDirectory references when content files lack the .cf extension by @SkowronskiAndrew in #105
• [#99] Add --build-history option to analyze by @SkowronskiAndrew in #109

Schema differences

Minor changes is case sensitivity and enforcing uniqueness for archive file names.

New "content_layout_*" tables are added.

New Contributors

• @Abhinav-0311 made their first contribution in #91

Full Changelog: v2.0.0...v2.1.0

v2.0.0 Analyze Schema Improvements

Overview

This version gets a major version bump because the schema for the sqlite database has some improvements, that involved changing the schema, so this is a "breaking change". If you have SQL queries or other integration based on analyze databases then you may need to update them when you upgrade from v1.X to v2.X versions.

This version does not itself contain major new features, but it has some important fixes and prepares for upcoming work on analyzing content directory builds. The change in #44 is a substantial size improvement n the analyze database. There are also improvements to how scenes dependencies are represented. And the code is better generalized for representing non-AssetBundle content. It is recommended to upgrade even if you are only analyzing AssetBundle builds.

What's Changed

• [#44] Normalize refs table to deduplicate property strings by @SkowronskiAndrew in #80
• [#82] Rename assets/asset_dependencies tables to reflect AssetBundle/PreloadData origin by @SkowronskiAndrew in #84
• [#81] Fix scene/preload dependencies for Player and Scriptable Build Pipeline builds by @SkowronskiAndrew in #87
• Document project structure; split Archive and SerializedFile into their own libraries by @SkowronskiAndrew in #88
• [#68] Generalize AssetBundle to Archive in analyze schema and code by @SkowronskiAndrew in #89
• [#55] Show type name in build_report_packed_asset_contents_view by @SkowronskiAndrew in #90

Schema differences

If you have queries or tooling built on a pre-2.0 analyze database, use the rename maps below to update them.

1. Table / view renames

Archive generalization (issue #68)

UnityDataTools now analyzes more than AssetBundles (Player, Content Archive, ContentDirectory builds), so AssetBundle-centric names for the generic archive concept were generalized. Names genuinely specific to the AssetBundle object were intentionally left unchanged.

Before	After
asset_bundles (table)	archives
serialized_files.asset_bundle	serialized_files.archive
object_view.asset_bundle (and *_asset_bundle aliases: material_/shader_/texture_/dep_)	object_view.archive (and *_archive)
in_bundles	in_archives
build_report_archive_contents.assetbundle	build_report_archive_contents.archive
build_report_archive_contents.assetbundle_content	build_report_archive_contents.archive_content

AssetBundle/PreloadData table clarification (issue #82)

The vaguely-named assets/asset_dependencies tables were renamed to reflect what actually populates them (the AssetBundle object's m_Container/m_PreloadTable and PreloadData). They are mostly empty for Player/ContentDirectory builds.

Before	After
assets (table)	assetbundle_assets
asset_view (view)	assetbundle_asset_view
asset_dependencies (table)	preload_dependencies
asset_dependencies_view (view)	preload_dependencies_view
index asset_dependencies_object	preload_dependencies_object
index asset_dependencies_dependency	preload_dependencies_dependency

Unchanged (genuinely AssetBundle-object-specific): assetbundle_assets, assetbundle_asset_view, preload_dependencies, AssetBundleHandler.

2. Human-readable types in build reports (issue #55)

• build_report_packed_asset_contents_view.type now shows the type name (e.g. Texture2D) instead of the numeric Class ID.
• The underlying build_report_packed_asset_info.type column still stores the numeric class id.

Full Changelog: v1.3.6...v2.0.0

v1.3.6 - UnityFileSystemApi update, find-refs fix, test coverage

What's Changed

• [#72] Add find-refs test coverage and fix its database/null-ref bugs by @SkowronskiAndrew in #79

• find-refs command fixed and now supports --stdout

• Update UnityFileSystemApi to 6000.6.0a7 so that UnityDataTools works with built output from Unity 6.6. It continues to support output from older versions of Unity.

• Testing: Reorganization of existing Unity project and add a second LeadingEdge Unity project that uses newer feature, e.g. serialized dictionaries and content directory builds. Resulting builds are added to source control and now used to test the dump and find-refs command. More usage of this test data will come in follow work.

Full Changelog: v1.3.5...v1.3.6

v1.3.5 - Usability improvements, initial Unity 6.6 support

What's Changed

Performance and CRC correctness

#70 includes a significant performance fix related to the calculation of CRCs. There are also correctness fixes for the CRC calculation. However investigation discovered another existing flaw with CRC that could be relevant when comparing objects between different analyze databases, That will require its own separate fix, tracked in #74 .

Usability improvements

• Improve dump command: add --stdout, refactors, more tests by @SkowronskiAndrew in #69
• [#49] Improve analyze file specification, CLI help and docs improvements by @SkowronskiAndrew in #76
• [#77] Detect missing TypeTrees up front in analyze and dump by @SkowronskiAndrew in #78

The support for "--stdout" makes it easier to use the dump command interactively and for piping in scripts.

The new support for file and path specification for analyze makes it easier to analyze individual files, and to analyze Build Report files along with the build output.

Initial support for content directory builds

New type being introduced in Unity 6.6. More support will come in follow up releases.

• [#70] Fix analyze CRC for cah:/ ContentDirectory resources; add --skip-crc by @SkowronskiAndrew in #73
• Document ContentLayout.json and publish its schema by @SkowronskiAndrew in #75

Documentation improvement

• Docs: Addressables local→remote dependency example by @timt-unity3d in #67
• Also doc improvements included in #77

Compatibility

These changes should be pretty backward compatible for existing scripts and tools based on past version of UnityDataTool.
However the fixes for CRC included in #70 can mean that the calculated CRC values for objects will now be different from the value calculated on the same objects in the older version of UnityDataTool.

Full Changelog: v1.3.4...v1.3.5

v1.3.4

What's Changed

• Add type filtering to dump command (Issue #52) by @Ijtihed in #63
• Improve archive Command and update UnityFileSystemAPI by @SkowronskiAndrew in #62
• Remove unnecessary files from the release zip files (Issue #61)

New Contributors

• @Ijtihed made their first contribution in #63

Full Changelog: v1.3.3...v1.3.4

v1.3.3 - Support for Extracted TypeTrees

What's Changed

• Added support for specifying an extracted type tree data file #60. This permits use of UnityDataTool to analyze the output of the new feature for reducing build size (available starting in Unity 6.5 + Addressables 2.9). See Addressable AssetBundle memory considerations for details.
• Update UnityFileSystem (dlls, so and dylib) to the version shipped with 6000.5.0a8 - this version supports version 23 of the serialized file format and has the API needed to load extracted type tree files.

New Contributors

• @paulb-unity made their first contribution in #60

Full Changelog: 1.3.2...v1.3.3

1.3.2

What's Changed

• serializedFile metadata command (Unity version, platform and typetree info) #59 The "-f json" version prints out a lot more details than the text version.
• serializedfile objectlist and externalrefs subcommands now work for files that don't have typetrees. Rather than loading the file with UnityFileSystem the serialized file is parsed directly.
• Clearer error messages for Analyze in some cases when UnityDataTool fails to open a file because it does not have TypeTrees.
• New UnityBinaryFormat assembly with the code to recognize Unity file formats and to parse the header/metadata section of a binary SerializedFile

Full Changelog: v1.3.1...1.3.2

v1.3.1

What's Changed

• Improve Error Handling with Serialized File Detection by #58
  • Archives, YAML files detected with clear error messages
  • Reliable way to detect if a file is a Unity SerializedFile despite lack of a distinctive file extension or "magic prefix"
  • Added serialized-file header subcommand for looking at the header of serialized files
  • Added testing and examples of different versions of the serialized file format.
• Addressables BuildLayout.json Analyze fixes by #56
• Fix error handling and reporting in analyze command (Issue #48) by #57

Full Changelog: v1.3.0...v1.3.1

v1.3.0

What's Changed

• Analyze command: MonoScript and BuildReport support
• New serialized-file command
• Doc improvements

See #46 for details of new functionality.

Full Changelog: v1.2.1...v1.3.0

v1.2.1

This release has the same functionality as 1.2.0

But it contains a fix for an SQL statement that was discovered when fixing the unit tests, and that might also cause problems when using the v1.2.0 build. See #39 for details.

It also includes improved test coverage, and cleanup in the source code white space and minor "style" cleanup.