Skip to content

Commit

Permalink
feat: support concurrency limit
Browse files Browse the repository at this point in the history
When enabling multi-file upload, you can control the concurrency by providing the `concurrencyLimit` property.

The `concurrencyLimit` property creates a `ConcurrencyRequester` instance, storing all requests in the instance's queue. The concurrent upload task quantity is restricted by the `concurrencyLimit` when using the `send` method of the instance at the end of the `uploadFiles` process.
  • Loading branch information
thep0y committed Jan 13, 2024
1 parent 6027d4c commit d5818b1
Show file tree
Hide file tree
Showing 5 changed files with 261 additions and 73 deletions.
72 changes: 35 additions & 37 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ online example: https://upload.react-component.vercel.app/

## Feature

* support IE11+, Chrome, Firefox, Safari
- support IE11+, Chrome, Firefox, Safari

## install

Expand All @@ -54,29 +54,30 @@ React.render(<Upload />, container);

### props

|name|type|default| description|
|-----|---|--------|----|
|name | string | file| file param post to server |
|style | object | {}| root component inline style |
|className | string | - | root component className |
|disabled | boolean | false | whether disabled |
|component | "div"|"span" | "span"| wrap component name |
|action| string &#124; function(file): string &#124; Promise&lt;string&gt; | | form action url |
|method | string | post | request method |
|directory| boolean | false | support upload whole directory |
|data| object/function(file) | | other data object to post or a function which returns a data object(a promise object which resolve a data object) |
|headers| object | {} | http headers to post, available in modern browsers |
|accept | string | | input accept attribute |
|capture | string | | input capture attribute |
|multiple | boolean | false | only support ie10+|
|onStart | function| | start upload file |
|onError| function| | error callback |
|onSuccess | function | | success callback |
|onProgress | function || progress callback, only for modern browsers|
|beforeUpload| function |null| before upload check, return false or a rejected Promise will stop upload, only for modern browsers|
|customRequest | function | null | provide an override for the default xhr behavior for additional customization|
|withCredentials | boolean | false | ajax upload with cookie send |
|openFileDialogOnClick | boolean | true | useful for drag only upload as it does not trigger on enter key or click event |
| name | type | default | description |
| --- | --- | --- | --- |
| name | string | file | file param post to server |
| style | object | {} | root component inline style |
| className | string | - | root component className |
| disabled | boolean | false | whether disabled |
| component | "div" &#124; "span" | "span" | wrap component name |
| action | string &#124; function(file): string &#124; Promise&lt;string&gt; | | form action url |
| method | string | post | request method |
| directory | boolean | false | support upload whole directory |
| data | object/function(file) | | other data object to post or a function which returns a data object(a promise object which resolve a data object) |
| headers | object | {} | http headers to post, available in modern browsers |
| accept | string | | input accept attribute |
| capture | string | | input capture attribute |
| multiple | boolean | false | only support ie10+ |
| concurrencyLimit | number &#124; undefined | undefined | asynchronously posts files with the concurrency limit |
| onStart | function | | start upload file |
| onError | function | | error callback |
| onSuccess | function | | success callback |
| onProgress | function | | progress callback, only for modern browsers |
| beforeUpload | function | null | before upload check, return false or a rejected Promise will stop upload, only for modern browsers |
| customRequest | function | null | provide an override for the default xhr behavior for additional customization |
| withCredentials | boolean | false | ajax upload with cookie send |
| openFileDialogOnClick | boolean | true | useful for drag only upload as it does not trigger on enter key or click event |

#### onError arguments

Expand All @@ -88,26 +89,23 @@ React.render(<Upload />, container);

1. `result`: response body
2. `file`: upload file
3. `xhr`: xhr header, only for modern browsers which support AJAX upload. since
2.4.0

3. `xhr`: xhr header, only for modern browsers which support AJAX upload. since 2.4.0

### customRequest

Allows for advanced customization by overriding default behavior in AjaxUploader. Provide your own XMLHttpRequest calls to interface with custom backend processes or interact with AWS S3 service through the aws-sdk-js package.

customRequest callback is passed an object with:

* `onProgress: (event: { percent: number }): void`
* `onError: (event: Error, body?: Object): void`
* `onSuccess: (body: Object): void`
* `data: Object`
* `filename: String`
* `file: File`
* `withCredentials: Boolean`
* `action: String`
* `headers: Object`

- `onProgress: (event: { percent: number }): void`
- `onError: (event: Error, body?: Object): void`
- `onSuccess: (body: Object): void`
- `data: Object`
- `filename: String`
- `file: File`
- `withCredentials: Boolean`
- `action: String`
- `headers: Object`

### methods

Expand Down
22 changes: 19 additions & 3 deletions src/AjaxUploader.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import type {
import defaultRequest from './request';
import traverseFileTree from './traverseFileTree';
import getUid from './uid';
import ConcurrencyRequester from './concurrencyRequest';

interface ParsedFileInfo {
origin: RcFile;
Expand All @@ -26,6 +27,8 @@ class AjaxUploader extends Component<UploadProps> {

reqs: any = {};

private concurrencyRequester?: ConcurrencyRequester<any>;

private fileInput: HTMLInputElement;

private _isMounted: boolean;
Expand Down Expand Up @@ -111,17 +114,26 @@ class AjaxUploader extends Component<UploadProps> {
return this.processFile(file, originFiles);
});

const { onBatchStart, concurrencyLimit } = this.props;

if (concurrencyLimit) {
this.concurrencyRequester = new ConcurrencyRequester(concurrencyLimit);

Check warning on line 120 in src/AjaxUploader.tsx

View check run for this annotation

Codecov / codecov/patch

src/AjaxUploader.tsx#L120

Added line #L120 was not covered by tests
}

// Batch upload files
Promise.all(postFiles).then(fileList => {
const { onBatchStart } = this.props;

onBatchStart?.(fileList.map(({ origin, parsedFile }) => ({ file: origin, parsedFile })));

fileList
.filter(file => file.parsedFile !== null)
.forEach(file => {
this.post(file);
});

// Asynchronously posts files with the concurrency limit.
if (this.concurrencyRequester) {
this.concurrencyRequester.send();

Check warning on line 135 in src/AjaxUploader.tsx

View check run for this annotation

Codecov / codecov/patch

src/AjaxUploader.tsx#L135

Added line #L135 was not covered by tests
}
});
};

Expand Down Expand Up @@ -230,7 +242,11 @@ class AjaxUploader extends Component<UploadProps> {
};

onStart(origin);
this.reqs[uid] = request(requestOption);
if (this.concurrencyRequester) {
this.reqs[uid] = this.concurrencyRequester.append(requestOption);

Check warning on line 246 in src/AjaxUploader.tsx

View check run for this annotation

Codecov / codecov/patch

src/AjaxUploader.tsx#L246

Added line #L246 was not covered by tests
} else {
this.reqs[uid] = request(requestOption);
}
}

reset() {
Expand Down
154 changes: 154 additions & 0 deletions src/concurrencyRequest.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
import { ConcurrencyRequestTask, UploadRequestOption } from './interface';
import { prepareData, prepareXHR } from './request';

/**
* Asynchronously processes an array of items with a concurrency limit.
*
* @template T - Type of the input items.
* @template U - Type of the result of the asynchronous task.
*
* @param {number} concurrencyLimit - The maximum number of asynchronous tasks to execute concurrently.
* @param {T[]} items - The array of items to process asynchronously.
* @param {(item: T) => Promise<U>} asyncTask - The asynchronous task to be performed on each item.
*
* @returns {Promise<U[]>} - A promise that resolves to an array of results from the asynchronous tasks.
*/
async function asyncPool<T, U>(
concurrencyLimit: number,
items: T[],
asyncTask: (item: T) => Promise<U>,
): Promise<U[]> {
const tasks: Promise<U>[] = [];
const pendings: Promise<U>[] = [];

Check warning on line 22 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L20-L22

Added lines #L20 - L22 were not covered by tests

for (const item of items) {
const task = asyncTask(item);
tasks.push(task);

Check warning on line 26 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L24-L26

Added lines #L24 - L26 were not covered by tests

if (concurrencyLimit <= items.length) {
task.then(() => {
pendings.splice(pendings.indexOf(task), 1);

Check warning on line 30 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L29-L30

Added lines #L29 - L30 were not covered by tests
});
pendings.push(task);

Check warning on line 32 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L32

Added line #L32 was not covered by tests

if (pendings.length >= concurrencyLimit) {
await Promise.race(pendings);

Check warning on line 35 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L35

Added line #L35 was not covered by tests
}
}
}

return Promise.all(tasks);

Check warning on line 40 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L40

Added line #L40 was not covered by tests
}

type DataType = 'form' | 'blob' | 'string';

/**
* Represents a class for handling concurrent requests with a specified concurrency limit.
*
* @template T - The type of data to be uploaded.
*/
export default class ConcurrencyRequester<T> {
/**
* The concurrency limit for handling requests simultaneously.
*/
private concurrencyLimit: number;

/**
* An array to store the tasks for concurrent requests.
*/
private tasks: ConcurrencyRequestTask[] = [];

Check warning on line 59 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L59

Added line #L59 was not covered by tests

/**
* The type of data to be sent in the request ('form', 'blob', or 'string').
*/
private dataType: DataType;

/**
* Creates an instance of ConcurrencyRequester.
*
* @param {number} concurrencyLimit - The concurrency limit for handling requests simultaneously.
* @param {DataType} [dataType='form'] - The type of data to be sent in the request ('form', 'blob', or 'string').
*/
constructor(concurrencyLimit: number, dataType: DataType = 'form') {
this.concurrencyLimit = concurrencyLimit;
this.dataType = dataType;

Check warning on line 74 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L73-L74

Added lines #L73 - L74 were not covered by tests
}

/**
* Prepares data based on the specified data type.
*
* @param {UploadRequestOption<T>} option - The upload request option.
* @returns {string | Blob | FormData} - The prepared data based on the specified data type.
* @private
*/
private prepareData = (option: UploadRequestOption<T>): string | Blob | FormData => {

Check warning on line 84 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L84

Added line #L84 was not covered by tests
if (this.dataType === 'form') {
return prepareData(option);

Check warning on line 86 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L86

Added line #L86 was not covered by tests
}

return option.file;

Check warning on line 89 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L89

Added line #L89 was not covered by tests
};

/**
* Prepares a task for a concurrent request.
*
* @param {UploadRequestOption<T>} option - The upload request option.
* @returns {ConcurrencyRequestTask} - The prepared task for the concurrent request.
* @private
*/
private prepare = (option: UploadRequestOption<T>): ConcurrencyRequestTask => {
const xhr = prepareXHR(option);

Check warning on line 100 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L99-L100

Added lines #L99 - L100 were not covered by tests

const data = this.prepareData(option);

Check warning on line 102 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L102

Added line #L102 was not covered by tests

const task: ConcurrencyRequestTask = { xhr, data };

Check warning on line 104 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L104

Added line #L104 was not covered by tests

xhr.onerror = function error(e) {
task.done?.();
xhr.onerror(e);

Check warning on line 108 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L106-L108

Added lines #L106 - L108 were not covered by tests
};

xhr.onload = function onload(e) {
task.done?.();
xhr.onload(e);

Check warning on line 113 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L111-L113

Added lines #L111 - L113 were not covered by tests
};

return task;

Check warning on line 116 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L116

Added line #L116 was not covered by tests
};

/**
* Appends a new upload request to the tasks array.
*
* @param {UploadRequestOption<T>} option - The upload request option.
* @returns {{ abort: () => void }} - An object with an `abort` function to cancel the request.
*/
append = (option: UploadRequestOption<T>): { abort: () => void } => {
const task = this.prepare(option);

Check warning on line 126 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L125-L126

Added lines #L125 - L126 were not covered by tests

this.tasks.push(task);

Check warning on line 128 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L128

Added line #L128 was not covered by tests

return {
abort() {
task.xhr.abort();

Check warning on line 132 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L130-L132

Added lines #L130 - L132 were not covered by tests
},
};
};

/**
* Sends all the appended requests concurrently.
*/
send = (): void => {
asyncPool(

Check warning on line 141 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L140-L141

Added lines #L140 - L141 were not covered by tests
this.concurrencyLimit,
this.tasks,
item =>
new Promise<void>(resolve => {
const xhr = item.xhr;

Check warning on line 146 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L145-L146

Added lines #L145 - L146 were not covered by tests

item.done = resolve;

Check warning on line 148 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L148

Added line #L148 was not covered by tests

xhr.send(item.data);

Check warning on line 150 in src/concurrencyRequest.ts

View check run for this annotation

Codecov / codecov/patch

src/concurrencyRequest.ts#L150

Added line #L150 was not covered by tests
}),
);
};
}
7 changes: 7 additions & 0 deletions src/interface.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ export interface UploadProps
input?: React.CSSProperties;
};
hasControlInside?: boolean;
concurrencyLimit?: number;
}

export interface UploadProgressEvent extends Partial<ProgressEvent> {
Expand Down Expand Up @@ -76,3 +77,9 @@ export interface UploadRequestOption<T = any> {
export interface RcFile extends File {
uid: string;
}

export interface ConcurrencyRequestTask {
xhr: XMLHttpRequest;
data: File | FormData | string | Blob;
done?: () => void;
}
Loading

0 comments on commit d5818b1

Please sign in to comment.