This article was co-authored with generative AI. Facts have been checked against public documentation where feasible, but errors may remain. Please verify primary sources before relying on this for important decisions.

In Archivematica, you can freely rearrange (arrange) a set of files, once sent to the backlog, into folders on the Appraisal / Arrangement tab, and ingest that hierarchy as-is bundled into a single SIP (Submission Information Package).

At this point a naive question arises.

Once you rearrange, is "what the original structure was" lost? And where, and in what form, is the "rearranged structure" recorded?

To state the conclusion first: both the original structure and the rearranged structure survive, without either being lost. However, the location and format of the recording differ between "the SIP stage" and "the AIP stage," so I'll separate and organize that with real data.

The verification environment is a development Docker stack on an Apple Silicon Mac (equivalent to Archivematica v1.18.0). The scenario I confirmed: put a transfer called flat-demo — in which notes.doc / report.pdf / scan.tif sit flatly, three of them, directly under objects/ — into the backlog, then fold it into folders via arrange.

I also made a demo video of this operation of "arranging flat material on the Appraisal tab into one SIP" (the audio and subtitles are auto-generated by generative AI).

Prerequisites: terminology and the overall flow

  • Transfer: the first unit of ingest. After format identification and virus check, it holds its own dedicated transfer METS. This is the unit that enters the backlog.
  • SIP: the working unit of Ingest. Created by arranging from the backlog.
  • AIP: the preservation package. Near the end of Ingest, the AIP METS is generated, packed into a .7z or the like, and stored in the Storage Service.

Here are the key points in a table first.

Before rearrangingAfter rearranging
SIP stage / DBmain_siparrange.original_pathmain_siparrange.arrange_path (working / non-persistent)
SIP stage / fileThe original transfer METS copied into the SIP (permanent)The SIP's physical directory structure
AIP stage / METSsubmissionDocumentation/transfer-…/METS.xml (bundles the original transfer METS)The AIP METS structMap (physical + logical)

Below, I look at the real data in order, SIP → AIP.

The SIP stage: where does the structure remain?

An important prerequisite: a dedicated METS that points only at the "SIP" object (a SIP-specific file distinct from METS.<SIPUUID>.xml) is not generated.

That said — and this is easy to misunderstand — the METS that contains the structMap is already created at the Transfer stage. Near the end of Transfer, createTransferMETS_v1.0 (client script create_transfer_mets.py, using the library metsrw) generates metadata/submissionDocumentation/METS.xml, which already contains the structMaps (physical Archivematica default and logical Normative Directory Structure). Checking on a real AIP, I found that this transfer METS CREATEDATE is about one hour earlier than the AIP METS (it exists first).

Meanwhile, what Ingest's create_mets_v2.0 (client script create_mets_v2.py) generates is the AIP METS (data/METS.<SIPUUID>.xml), and this is the first time the "rearranged structure" is recorded as a structMap. In other words, it is not that "there is no structMap at all at the SIP stage" — rather, turning the rearranged structure into a structMap happens at the timing of AIP METS generation.

So where is "before/after rearranging" at the SIP stage? There are two lines.

① The before/after mapping → the DB main_siparrange table

The "original path → rearranged path" at the moment you arrange and create the SIP goes into this table.

SELECT original_path, arrange_path, sip_created, aip_created, HEX(file_uuid) AS file_uuid
FROM MCP.main_siparrange ORDER BY arrange_path;

The actual contents (the three files of flat-demo; here, the case where each file is placed directly at the root):

original_path (before = the location in the backlog)arrange_path (after = the location within the SIP)file_uuid
originals/flat-demo-12cec7db…/data/objects/notes.docnotes.doca565f459…
…/objects/report.pdfreport.pdfa063d929…
…/objects/scan.tifscan.tiff45e5c3a…
  • original_path … before rearranging (the path in the source backlog)
  • arrange_path … after rearranging (the relative path within the SIP). If you distributed them into folders a/ b/ c/ and bundled them into one SIP, this would become a/notes.doc / b/report.pdf / c/scan.tif.
  • file_uuid … the file UUID. This is identical to the AIP METS FILEID="file-a565f459-…" downstream, serving as an invariant key that runs through SIP→AIP.

:::message alert main_siparrange is operational working data, not a permanent provenance record. You can delete it with DELETE, and when redoing an arrange, it is sometimes actually cleared and re-entered. Therefore the body of the "permanently surviving before/after rearranging structure" is the next ②, and the AIP METS described later. :::

② The pre-rearrange structure → the original transfer METS copied into the SIP (permanent)

When the SIP is created, the original transfer's METS is copied into the SIP.

<SIP>/objects/submissionDocumentation/transfer-flat-demo-12cec7db…/METS.xml

Inside this original transfer METS, the flat structure before rearranging remains as-is (real data):

<mets:structMap TYPE="physical" LABEL="Archivematica default">
  <mets:div LABEL="flat-demo-12cec7db-…">
    <mets:div LABEL="objects">
      <mets:div TYPE="Item" LABEL="notes.doc"/>
      <mets:div TYPE="Item" LABEL="report.pdf"/>   <!-- originally three, laid out flatly -->
      <mets:div TYPE="Item" LABEL="scan.tif"/>

In other words, "the pre-rearrange form" is already bundled inside the SIP at the SIP stage, and is carried over into the AIP as-is.

③ The rearranged structure → the SIP's physical directory itself

The SIP is physically assembled as "the directory resulting from the arrange" (objects/a/ objects/b/ objects/c/ …; in implementation, at arrange confirmation the files are actually shutil.moved with arrange_path as the destination). But, to repeat, this rearranged layout becomes a record in the form of a METS structMap only from Ingest's create_mets_v2.0 onward; at the SIP stage it merely exists as a structure on the filesystem (as noted, the METS containing the structMap already exists as the transfer METS, but what it represents is the structure before rearranging).

The AIP stage: how is the structure fixed?

As Ingest proceeds, create_mets_v2.0 generates the AIP METS (data/METS.<SIPUUID>.xml), and here the structure is formally recorded as a METS structMap. Typically the following two kinds of structMap are written.

:::message Strictly, it is not "always two kinds." The physical Archivematica default is always written, but the logical Hierarchical is output only for a SIP that was rearranged via SIP Arrange (if there are no SIPArrange rows, build_arranged_structmap() returns None). Conversely, depending on settings, another logical structMap such as Normative Directory Structure may be added. This article assumes the case of "arranged and bundled into one SIP." :::

After rearranging (physical) → structMap TYPE="physical"

The actual layout on disk, i.e., the result of the arrange rearrangement.

<mets:structMap TYPE="physical" LABEL="Archivematica default">
  <mets:div TYPE="Directory" LABEL="objects">
    <mets:div TYPE="Item" LABEL="notes.doc">
      <mets:fptr FILEID="file-a565f459-…"/>

If you distributed them into a/ b/ c/ and bundled them into one SIP, this becomes nested divs like objects/a/notes.doc.

After rearranging (intellectual structure) → structMap TYPE="logical" LABEL="Hierarchical"

The hierarchy you built in the Arrangement pane is recorded independently of the physical paths.

<mets:structMap TYPE="logical" LABEL="Hierarchical">
  <mets:div LABEL="objects">
    <mets:div LABEL="notes.doc">
      <mets:fptr FILEID="file-a565f459-…"/>

Before rearranging → the original transfer METS bundled inside the AIP (permanent)

The original transfer METS bundled at the SIP stage remains inside the AIP as-is.

data/objects/submissionDocumentation/transfer-flat-demo-12cec7db…/METS.xml

This makes "how things were laid out before the arrange" permanently traceable from the AIP alone.

The link connecting before ↔ after → the file UUID and PREMIS originalName

  • The same file UUID appears in both the original transfer METS and the AIP METS, so per file you can trace "where it was → where it is now" (the METS FILEID attribute has the form file-<UUID>. The bare UUID is the invariant key in the DB = the PREMIS objectIdentifierValue, and FILEID is that with file- prefixed). Note that a preservation derivative has a new UUID and appears only in the AIP METS. What is linked is the original files to each other.

  • Each file's PREMIS carries an originalName (real data):

    <premis:originalName>%transferDirectory%objects/notes.doc</premis:originalName>

    No matter where you move it physically, the original path at ingest time remains in PREMIS.

  • Furthermore, a log of the rearrange operations remains in data/logs/arrange.log.

Summary

  • "Rearranging does not lose the original structure." The design has before and after coexist in the same package, cross-linked by file UUID.
  • At the SIP stage,
    • the before/after correspondence is the DB main_siparrange (but working / non-persistent),
    • the before structure is the original transfer METS copied into the SIP (permanent),
    • the after structure is the SIP's physical directory itself,
    • a "dedicated METS pointing only at the SIP object" is not generated (but the METS containing the structMap itself already exists as the transfer METS at the Transfer stage, and what it represents is the before structure).
  • Only at the AIP stage is the after structure fixed into the AIP METS as a structMap (physical Archivematica default +, if arranged, logical Hierarchical), and the before structure is retained as the bundled original transfer METS.

For the worry "does provenance survive even after rearranging?", it becomes easier to follow the behavior if you understand it as: the rearranged structure takes its final form as a METS structMap at the timing of AIP generation (create_mets_v2.0), where it settles into physical + logical structMaps + the bundled original transfer METS.