partdec

package module
v0.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 3, 2025 License: Apache-2.0 Imports: 20 Imported by: 0

README

partdec

Go Report Card Codacy Badge

partdec is a command-line utility for multipart downloading and file splitting. It can download a file in parts simultaneously from a remote or local source and distribute parts of the file to multiple destination paths.

partdec allows a separate connection per file part and the ability to resume interrupted file transfers. It supports HTTP and HTTPS protocols.

Demo

0-demo

See more ...

Use Cases

  • Download Acceleration
    Multipart or segmented download is a commonly used technique to accelerate download speeds by retrieving multiple parts of a file simultaneously.

  • Data Partitioning and Distribution
    In cases when data has to come from one endpoint and then be split into parts and distributed across multiple nodes or servers.

  • Large File Management
    Transferring a file in parts solves the problem of file size restrictions imposed by file systems. For instance, transferring a file larger than 4GB to a FAT32 system.

Installation

[!NOTE] For installation, go version 1.22.8 or later is required.

Install it with go:

go install github.com/cjijcb/partdec/cmd/partdec@latest

Or, to build the binary, download the latest source code archive from the releases page, extract it, then run:

cd partdec/cmd/partdec/
go build -o <BINPATH> partdec.go  # Replace <BINPATH> with the path where the binary file should go

Usage Information

Basic Options:

  -p <N>     Split the file into N parts.
  -s <SIZE>  Split the file into parts of SIZE.
  -b <PATH>  Set the base path for output files and also set their filename.
  -d <PATH>  Set the destination directory for output files. PATH can be comma-separated directories.

See Full Usage Information.

Merging Files

You can use cat, a standard Unix utility, to merge files. Other similar applications can work as well. To merge files using cat, the paths must be passed in ascending order based on the numeric suffix of the files. You can also use a wildcard (*) to represent these numeric suffixes. As follows:

Sypnosis:

cat [PATH]<FILENAME>_* ... > [PATH]<NEWFILENAME>

Examples:

cat archive.zip_* > my_archive.zip

cat /tmp/archive.zip_* > ~/Downloads/my_archive.zip

cat /tmp/archive.zip_* /var/archive.zip_* > ~/Downloads/my_archive.zip

Development

It is still in early development, and a lot can still change. Any contributions are welcome. There are also plans to support other protocols such as FTP(S) and SFTP.

License

Copyright (C) 2024 Carlo Jay I. Jacaba

partdec is licensed under the Apache License, Version 2.0.

Documentation

Index

Constants

View Source 
const (
	File DLType = iota
	HTTP

	PartSoftLimit      = 128
	MaxConcurrentFetch = 32
)
View Source 
const (
	New FileState = iota
	Resume
	Completed
	Broken
	Unknown

	FilePerm os.FileMode = 0644

	UnknownSize   = -1
	PathSeparator = string(os.PathSeparator)
)
View Source 
const (
	Kibi = 1024
	Mebi = 1024 * 1024
	Gibi = 1024 * 1024 * 1024
	Tebi = 1024 * 1024 * 1024 * 1024

	Kilo = 1000
	Mega = 1000 * 1000
	Giga = 1000 * 1000 * 1000
	Tera = 1000 * 1000 * 1000 * 1000
)
View Source 
const (
	UserAgent = "partdec/0.4.2"
)

Variables

View Source 
var (
	JoinErr = errors.Join
	IsErr   = errors.Is
	NewErr  = fmt.Errorf

	Stderr = os.Stderr

	ErrCancel     = NewErr("canceled")
	ErrAbort      = NewErr("aborted")
	ErrPartExceed = NewErr("part total count or size exceeds the source file size")
	ErrFileURL    = NewErr("inaccessible file or invalid URI")
	ErrDLType     = NewErr("unknown download type")
	ErrExhaust    = NewErr("resource exhausted")
	ErrArgs       = NewErr("invalid argument")
	ErrParse      = NewErr("parse error")
	ErrPartLimit  = NewErr("exceeds output file count limit")
	ErrMultPart   = NewErr("server does not support multipart or segmented downloads")
	ErrRedir      = NewErr("redirected")
	ErrVer        = NewErr("version requested")
)
View Source 
var (
	SharedTransport = &http.Transport{
		MaxIdleConnsPerHost: MaxConcurrentFetch,
		DisableKeepAlives:   false,
	}

	SharedHeader = http.Header{
		"Accept":     []string{"*/*"},
		"User-Agent": []string{UserAgent},
	}
)
View Source 
var (
	ByteUnit = map[string]int64{
		"":  1,
		"B": 1,

		"KIB": Kibi,
		"MIB": Mebi,
		"GIB": Gibi,
		"TIB": Tebi,

		"K": Kibi,
		"M": Mebi,
		"G": Gibi,
		"T": Tebi,

		"KB": Kilo,
		"MB": Mega,
		"GB": Giga,
		"TB": Tera,
	}
)
View Source 
var HelpPage string
View Source 
var VersionPage string

Functions

func BuildRangeHeader 

func BuildRangeHeader(br ByteRange) string

func FileNameIndexer 

func FileNameIndexer(maxIndex int) func(string) string

func IsEndSeparator added in v0.1.1 

func IsEndSeparator(path string) bool

func IsFile 

func IsFile(path string) bool

func IsURL 

func IsURL(rawURL string) bool

func NewFileName 

func NewFileName(uri string, hdr http.Header) string

Generates a file name based on headers, URL, or file path.

func ShowProgress 

func ShowProgress(d *Download)

Types

type ByteRange 

type ByteRange struct {
	Start, End, Offset int64
	Indeterminate      bool
}

type DLOptions 

type DLOptions struct {
	URI       string
	BasePath  string
	DstDirs   []string
	PartCount int
	PartSize  int64
	ReDL      FileResets
	UI        func(*Download)
	Force     bool
	Mod       *IOMod
}

func NewDLOptions 

func NewDLOptions() (*DLOptions, error)

func (*DLOptions) AlignPartCountSize 

func (opt *DLOptions) AlignPartCountSize(dataSize int64) error

Adjusts the part count and size to fit the given data size.

func (*DLOptions) ParseBasePath added in v0.2.7 

func (opt *DLOptions) ParseBasePath(hdr http.Header)

Sets the base path for storing the file, appending the file name if the base path ends with a separator.

type DLType 

type DLType uint8

type DataCaster 

type DataCaster interface {
	DataCast(ByteRange) (io.ReadCloser, error)
	Close() error
	IsOpen() bool
}

Interface with methods for casting a range of bytes and managing the lifecycle of a data stream.

func NewFileDataCaster 

func NewFileDataCaster(path string) (DataCaster, error)

func NewHTTPDataCaster added in v0.2.7 

func NewHTTPDataCaster(rawURL string) (DataCaster, error)

type Download 

type Download struct {
	Files     FileIOs
	Sources   []DataCaster
	URI       string
	DataSize  int64
	Type      DLType
	UI        func(*Download)
	Resumable bool
	Mod       *IOMod
	Flow      *FlowControl
	Stop      context.CancelFunc
	Ctx       context.Context
}

Manages the entire download process.

func NewDownload 

func NewDownload(opt *DLOptions) (d *Download, err error)

Creates a new Download instance based on the provided DLOptions. It determines the download type (file or HTTP), initializes file I/O handlers, and sets up download configurations.

func (*Download) DataCasterGenerator 

func (d *Download) DataCasterGenerator() func() (DataCaster, error)

Creates a function that generates DataCasters for fetching data. It uses circular indexing to reuse existing DataCasters.

func (*Download) InitFiles added in v0.2.7 

func (d *Download) InitFiles(partSize int64, fr FileResets) (err error)

Initializes the file parts for download by setting their byte ranges and determining their initial state based on resumability

func (*Download) Start 

func (d *Download) Start() (err error)

Initiates the download process by setting up context handling, UI updates, and concurrent fetching of all file parts.

type FileIO 

type FileIO struct {
	*os.File
	Scope ByteRange
	State FileState
	Path  FilePath
	Oflag int
	Perm  os.FileMode
	// contains filtered or unexported fields
}

func NewFileIO 

func NewFileIO(base, dir string, oflag int) (*FileIO, error)

func (*FileIO) Close 

func (fio *FileIO) Close() error

func (*FileIO) DataCast 

func (fio *FileIO) DataCast(br ByteRange) (io.ReadCloser, error)

func (*FileIO) IsOpen 

func (fio *FileIO) IsOpen() bool

func (*FileIO) Open 

func (fio *FileIO) Open() (err error)

func (*FileIO) PullState 

func (fio *FileIO) PullState() FileState

func (*FileIO) PushState 

func (fio *FileIO) PushState(fs FileState)

func (*FileIO) SetOffset added in v0.4.1 

func (fio *FileIO) SetOffset() error

func (*FileIO) Size 

func (fio *FileIO) Size() (int64, error)

func (*FileIO) Truncate added in v0.4.1 

func (fio *FileIO) Truncate(size int64) error

type FileIOs 

type FileIOs []*FileIO

func BuildFileIOs 

func BuildFileIOs(partCount int, base string, dirs []string) (FileIOs, error)

func (FileIOs) RenewByState 

func (fios FileIOs) RenewByState(fr FileResets) error

func (FileIOs) SetByteRange 

func (fios FileIOs) SetByteRange(dataSize int64, partSize int64) error

func (FileIOs) SetInitialState added in v0.2.7 

func (fios FileIOs) SetInitialState() error

func (FileIOs) TotalSize 

func (fios FileIOs) TotalSize() int64

type FilePath 

type FilePath struct {
	Base, DstDir, Relative string
}

type FileResets added in v0.2.7 

type FileResets map[FileState]bool

func (*FileResets) Set added in v0.3.0 

func (fr *FileResets) Set(value string) error

func (*FileResets) String added in v0.3.0 

func (fr *FileResets) String() string

func (*FileResets) Type added in v0.3.0 

func (fr *FileResets) Type() string

type FileState 

type FileState uint8

func (FileState) String 

func (s FileState) String() string

type FlowControl 

type FlowControl struct {
	WG      *sync.WaitGroup
	Limiter chan struct{}
}

func NewFlowControl 

func NewFlowControl(limit int) *FlowControl

func (*FlowControl) Acquire 

func (fc *FlowControl) Acquire()

func (*FlowControl) Release 

func (fc *FlowControl) Release()

type HTTPIO added in v0.2.7 

type HTTPIO struct {
	*http.Client
	*http.Request
	Body io.ReadCloser
	// contains filtered or unexported fields
}

func NewHTTPIO added in v0.2.7 

func NewHTTPIO(ct *http.Client, rawURL string) (*HTTPIO, error)

func (*HTTPIO) Close added in v0.2.7 

func (hio *HTTPIO) Close() error

func (*HTTPIO) DataCast added in v0.2.7 

func (hio *HTTPIO) DataCast(br ByteRange) (io.ReadCloser, error)

func (*HTTPIO) IsOpen added in v0.2.7 

func (hio *HTTPIO) IsOpen() bool

type IOMod added in v0.3.0 

type IOMod struct {
	Retry       int
	Timeout     time.Duration
	UserHeader  http.Header
	NoConnReuse bool
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
partdec command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.