profile
viewpoint

klauspost/compress 1924

Optimized compression packages

klauspost/cpuid 432

CPU feature identification for Go

klauspost/asmfmt 179

Go Assembler Formatter

klauspost/dedup 175

Streaming Deduplication Package for Go

klauspost/crc32 73

CRC32 hash with x64 optimizations

klauspost/geoip-service 71

A fast in-memory http microservice for looking up MaxMind GeoIP2 and GeoLite2 database

klauspost/gad 41

Go After Dark

klauspost/doproxy 30

Reverse Proxy for managing multiple Digital Ocean backends.

klauspost/gfx 8

Graphic Drawing Library

klauspost/cld2 7

CLD2 (Compact Language Detector 2) bindings for Go (golang)

issue closedminio/sha256-simd

Implement encoding.BinaryMarshaler and encoding.BinaryUnmarshaler?

I think it would be even better if this project can implement encoding.BinaryMarshaler and encoding.BinaryUnmarshaler. And of course, the implementation should always be fully compatible with crypto/sha256 (for seamless migration). ❤️

closed time in an hour

aofei

pull request commentminio/minio

Fix replication regression

Mint Automation

Test Result
mint-large-bucket.sh :heavy_check_mark:
mint-fs.sh :heavy_check_mark:
mint-gateway-s3.sh :heavy_check_mark:
mint-erasure.sh :heavy_check_mark:
mint-dist-erasure.sh :heavy_check_mark:
mint-zoned.sh :heavy_check_mark:
mint-gateway-nas.sh :heavy_check_mark:
mint-gateway-azure.sh :x: more...

<a name="72c747e9-5181-4727-8dc5-c2218599de84"></a>

11356-941f750/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:31440
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp dbab526f51b4:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 11 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 3 minutes and 49 seconds
(4/15) Running aws-sdk-ruby tests ... done in 21 seconds
(5/15) Running awscli tests ... done in 3 minutes and 3 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 39 minutes and 37 seconds
(8/15) Running minio-dotnet tests ... done in 4 minutes and 12 seconds
(9/15) Running minio-go tests ... done in 26 minutes and 54 seconds
(10/15) Running minio-java tests ... FAILED in 9 minutes and 18 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 169,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:28a157f7-e01e-0050-65e3-f4f2e6000000\nTime:2021-01-27T19:35:15.5158279Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'Oy5KNJJ3h8jCwOLOofTNqfU5be1e/Lg/u6oOCJkEtHg=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:8beaa75b-2a04-467a-45d5-bb92a10c1f1d\nx-ms-date:Wed, 27 Jan 2021 19:35:15 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商品\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-1mil7kh/minio-java-test-1hilfrb\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-1mil7kh/minio-java-test-1hilfrb?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2021-01-27T18:01:26Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [8beaa75b-2a04-467a-45d5-bb92a10c1f1d]\n   X-Ms-Date: [Wed, 27 Jan 2021 19:35:15 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Wed, 27 Jan 2021 19:35:15 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [28a157f7-e01e-0050-65e3-f4f2e6000000]\n\n\n, bucketName = minio-java-test-1mil7kh, objectName = minio-java-test-1hilfrb, resource = /minio-java-test-1mil7kh/minio-java-test-1hilfrb, requestId = 165E2DD660417345, hostId = c4d4245d-28bd-4c36-a10d-58974a9be285)\nrequest={method=PUT, url=http://minio-c2.minio.io:31440/minio-java-test-1mil7kh/minio-java-test-1hilfrb, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-c2.minio.io:31440\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20210127T193515Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210127/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 165E2DD660417345\nX-Xss-Protection: 1; mode=block\nDate: Wed, 27 Jan 2021 19:35:15 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... done in 3 minutes and 56 seconds
(11/15) Running minio-py tests ... done in 22 minutes and 43 seconds
(12/15) Running s3cmd tests ... done in 9 minutes and 53 seconds
(13/15) Running s3select tests ... done in 1 minutes and 7 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub Deleting image locally

poornas

comment created time in 2 hours

issue closedminio/minio

MinIO: Disk not working in docker swarm mode

Hi, My minio standalone deployment works just fine with my setup but I'm having an issue with docker swarm mode deployment For the manager node I get this error: In the beginning of docker logs on the manager node, i get :

`Disk `/export1` is a root disk. Please ensure the disk is mounted properly, refusing to use  root disk.

For all the disks

API: SYSTEM()
Time: 06:07:56 UTC 01/18/2021
DeploymentID: 0ee2001b-a23b-44cf-ba3d-13a0c04059e0
Error: Write failed. Insufficient number of disks online (*errors.errorString)
      10: /go/minio/cmd/erasure-encode.go:100:cmd.(*Erasure).Encode()
       9: /go/minio/cmd/erasure-object.go:683:cmd.erasureObjects.putObject()
       8: /go/minio/cmd/erasure-object.go:578:cmd.erasureObjects.PutObject()
       7: /go/minio/cmd/erasure-sets.go:783:cmd.(*erasureSets).PutObject()
       6: /go/minio/cmd/erasure-server-pool.go:560:cmd.(*erasureServerPools).PutObject()
       5: /go/minio/cmd/config-common.go:67:cmd.saveConfig()
       4: /go/minio/cmd/config-encrypted.go:233:cmd.migrateConfigPrefixToEncrypted()
       3: /go/minio/cmd/config-encrypted.go:59:cmd.handleEncryptedConfigBackend()
       2: /go/minio/cmd/server-main.go:292:cmd.initServer()
       1: /go/minio/cmd/server-main.go:500:cmd.serverMain()
Waiting for all MinIO sub-systems to be initialized.. possible cause (Unable to migrate all config at .minio.sys/con>
Waiting for all MinIO sub-systems to be initialized.. lock acquired
Attempting encryption of all config, IAM users and policies on MinIO backend

On Worker nodes i get:

API: SYSTEM()
Time: 06:33:56 UTC 01/18/2021
DeploymentID: 0ee2001b-a23b-44cf-ba3d-13a0c04059e0
Error: Write failed. Insufficient number of disks online (*errors.errorString)
       6: /go/minio/cmd/erasure-encode.go:100:cmd.(*Erasure).Encode()
       5: /go/minio/cmd/erasure-object.go:683:cmd.erasureObjects.putObject()
       4: /go/minio/cmd/erasure-object.go:578:cmd.erasureObjects.PutObject()
       3: /go/minio/cmd/erasure-sets.go:783:cmd.(*erasureSets).PutObject()
       2: /go/minio/cmd/erasure-server-pool.go:560:cmd.(*erasureServerPools).PutObject()
       1: /go/minio/cmd/data-usage.go:54:cmd.storeDataUsageInBackend()
API: SYSTEM()
Time: 06:33:56 UTC 01/18/2021
DeploymentID: 0ee2001b-a23b-44cf-ba3d-13a0c04059e0
Error: Storage resources are insufficient for the write operation .minio.sys/tmp/32ce053a-f47a-4fb1-a81d-a99879d7e9e0/e7646217-f0fb-49e3-9617-6f0c324933a5/part.1 (cmd.InsufficientWriteQuorum)
       1: /go/minio/cmd/data-usage.go:56:cmd.storeDataUsageInBackend()

My docker file is here:

version: '3.8'
services:
  minio1:
    image: minio/minio:edge
    hostname: minio1
    volumes:
      - minio1-data1:/export1
      - minio1-data2:/export2
      - minio1-data3:/export3
      - minio1-data4:/export4
      - minio1-data5:/export5
      - minio1-data6:/export6
    ports:
      - "9001:9000"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio1==true
    command: server --address :9000 http://minio1{1...2}/export{1...6}
    secrets:
      - secret_key
      - access_key
  minio2:
    image: minio/minio:edge
    hostname: minio2
    volumes:
      - minio2-data1:/export1
      - minio2-data2:/export2
      - minio2-data3:/export3
      - minio2-data4:/export4
      - minio2-data5:/export5
      - minio2-data6:/export6
    ports:
      - "9002:9000"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio2==true
    command: server --address :9000 http://minio{1...2}/export{1...6}
    secrets:
      - secret_key
      - access_key
volumes:
  minio1-data1:
  minio1-data2:
  minio1-data3:
  minio1-data4:
  minio1-data5:
  minio1-data6:
  minio2-data1:
  minio2-data2:
  minio2-data3:
  minio2-data4:
  minio2-data5:
  minio2-data6: 
networks:
  minio_distributed:
    driver: overlay
    attachable: true
secrets:
  secret_key:
    external: true
  access_key:
    external: true

Your Environment

<!--- Include as many relevant details about the environment you experienced the bug in -->

  • Version used (minio --version): minio/edge
  • Server setup and configuration: 2 nodes 6 disks
  • Operating System and version (uname -a): Linux openml1 5.4.0-60-generic

closed time in 2 hours

prabhant

issue commentminio/minio

MinIO: Disk not working in docker swarm mode

Seems like disk formatting was the issue Thanks :)

prabhant

comment created time in 2 hours

issue commentminio/minio

MinIO: Disk not working in docker swarm mode

do your backend support O_DIRECT? @prabhant

prabhant

comment created time in 3 hours

issue closedminio/minio

Support high number of users in IAM

Expected Behavior

MinIO installed in gateway mode, is provisioned with >500 users + policies. After a restart of the MinIO server, the users and policies should be available for usage. (thanks to etcd backend)

Current Behavior

After a restart of the MinIO server, the users and policies are not available.

  • user could not login
  • mc admin user info [...] could not respond

Steps to Reproduce

https://gist.github.com/christophemorio/eb67efd9819e89f778617ef5c6c8d705

  1. start an etcd server
  2. start a MinIO server in s3 gateway mode
  3. run this script to provision many users
  4. restart MinIO
  5. query any user or policy => it returns a timeout, or item not found, with error logs on MinIO server

Console logs

2021-01-19T14:51:14.600580431Z 
2021-01-19T14:51:14.600615626Z  You are running an older version of MinIO released 1 week ago 
2021-01-19T14:51:14.600620578Z  Update: Run `mc admin update` 
2021-01-19T14:51:14.600624063Z 
2021-01-19T14:51:14.600627338Z 
2021-01-19T14:51:14.601016356Z Endpoint:  http://172.17.0.5:9000  http://127.0.0.1:9000
2021-01-19T14:51:14.601030536Z 
2021-01-19T14:51:14.601034637Z Browser Access:
2021-01-19T14:51:14.601038867Z    http://172.17.0.5:9000  http://127.0.0.1:9000
2021-01-19T14:51:14.601042281Z 
2021-01-19T14:51:14.601079931Z Object API (Amazon S3 compatible):
2021-01-19T14:51:14.601089242Z    Go:         https://docs.min.io/docs/golang-client-quickstart-guide
2021-01-19T14:51:14.601125754Z    Java:       https://docs.min.io/docs/java-client-quickstart-guide
2021-01-19T14:51:14.601375066Z    Python:     https://docs.min.io/docs/python-client-quickstart-guide
2021-01-19T14:51:14.601386686Z    JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
2021-01-19T14:51:14.601391063Z    .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
2021-01-19T14:51:43.830119487Z {"level":"warn","ts":"2021-01-19T14:51:43.829Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-ebbd6fcb-7268-4953-a1c7-135919bb5ec3/etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-01-19T14:51:43.83025429Z 
2021-01-19T14:51:43.830267145Z API: SYSTEM()
2021-01-19T14:51:43.830273081Z Time: 14:51:43 UTC 01/19/2021
2021-01-19T14:51:43.830290925Z DeploymentID: db2a4cf3-14bd-4e01-b26a-065aae081049
2021-01-19T14:51:43.830296712Z Error: Unable to initialize IAM sub-system, some users may not be available etcd is unreachable, please check your endpoints [http://etcd:2379] (*fmt.wrapError)
2021-01-19T14:51:43.830302529Z        1: cmd/iam.go:530:cmd.(*IAMSys).Init()

Context

Server RELEASE.2019-10-02T21-19-38Z in gateway mode used to works fine with high number of users and policies (one per user). The same users and policies fails to apply on RELEASE.2021-01-05T05-22-38Z after a restart of the server.

Regression

This was not an issue for old server prior to ~RELEASE.2021-01-05T05-22-38Z~ RELEASE.2019-10-12T01-39-57Z.

Your Environment

(test environment)

  • Version used (minio --version): docker.io/minio/minio:RELEASE.2021-01-05T05-22-38Z
  • Server setup and configuration: minikube
  • Operating System and version (uname -a): Linux minikube 4.19.114 #1 SMP Mon Aug 3 12:35:22 PDT 2020 x86_64 GNU/Linux
  • one etcd v3.4.14 (cluster of a single member)
  • one replicas of MinIO pod

closed time in 3 hours

christophemorio

issue commentminio/minio

Support high number of users in IAM

Closing this issue as it's sufficiently addressed to support large encrypted users - when the server is loading them background requests

  • API calls fetch the user directly from etcd as a fallback
  • UserInfo like calls also fetch directly from etcd as a fallback
  • ListUsers mc admin user list might not finish quickly as the server loading them
    • clients are advised not to do it and let load them properly in background
christophemorio

comment created time in 3 hours

issue closedminio/minio

Bucket Notification Guide AMQP not support ssl

when AMQP config ssl , how to configure the configuration file to notification

i don`t find the relevant configuration properties

closed time in 3 hours

kkangkang789

issue commentminio/minio

Bucket Notification Guide AMQP not support ssl

No it does not @kkangkang789 and there are no plans - feel free to send a PR

kkangkang789

comment created time in 3 hours

issue openedminio/mc

mirror with --watch parameter: "Unable to list comparison retrying.. XML syntax error on line 2: expected attribute name in element"

While possible to copy files and also starting to mirror a directory to a bucket, mc fails to mirror with --watch parameter, throwing the following error:

Unable to list comparison retrying.. XML syntax error on line 2: expected attribute name in element

Expected Behavior

mc should mirror the directory and "watch" it.

Current Behavior

mc is not mirroring the directory, but instead looping an error message "XML syntax error on line 2: expected attribute name in element".

Possible Solution

If there are issues with any files or whatever the reason is to be unable to mirror, it should point me to the problem.

Steps to Reproduce (for bugs)

For me: mc mirror --watch /data myminio/backup

Context

Trying to backup user files to a Minio server.

Regression

Yes. Backups do not work, and problem cannot fixed without knowing the reason.

Your Environment

  • Version used (minio --version):
    • Server: minio version RELEASE.2021-01-16T02-19-44Z
    • Client: mc version RELEASE.2021-01-16T02-45-34Z
  • Server setup and configuration: Minio cluster consisting of 8 nodes
  • Operating System and version (uname -a):
    • Server:
      • Linux fnf-minio-08 5.4.0-64-generic minio/minio#72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
      • Ubuntu 20.04.1 LTS
    • Client:
      • Linux FNF-Host-MM-YA 4.15.0-134-generic minio/minio#138-Ubuntu SMP Fri Jan 15 10:52:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
      • Ubuntu 18.04.5 LTS

created time in 3 hours

push eventminio/minio

Poorna Krishnamoorthy

commit sha fd3f02637acf918a2db2bebede7f8bc99c1b9201

fix: replication regression due to proxying requests (#11356) In PR #11165 due to incorrect proxying for 2 way replication even when the object was not yet replicated Additionally, fix metadata comparisons when deciding to do full replication vs metadata copy. fixes #11340

view details

push time in 3 hours

PR merged minio/minio

Reviewers
Fix replication regression priority: high

From PR #11165 due to incorrect proxying for 2 way replication even when the object was not yet replicated

Also fixing metadata comparisons when deciding to do full replciation vs metadata copy.

Also Fixes #11340

Description

Motivation and Context

How to test this PR?

Turn on 2 way replication and do some uploads to source. mc admin trace should show a PUT operation on the dest, not a COPY in master due to incorrect proxying.

Types of changes

  • [x] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Optimization (provides speedup with no functional changes)
  • [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • [x] Fixes a regression (If yes, please add commit-id or PR # here)
  • [ ] Documentation updated
  • [ ] Unit tests added/updated
+76 -33

1 comment

8 changed files

poornas

pr closed time in 3 hours

issue closedminio/minio

File replication stopped work for bucket where enabled DeleteReplication and DeleteMarkerReplication

<!--- Provide a general summary of the issue in the Title above --> I have buckets in the two datacenter with enabled replication between each other. Replication of files stopped working after enabling DeleteReplication and DeleteMarkerReplication feature.

mc replicate edit local/test --id "bt8u4bnckkkgn5mck280" --replicate "delete,delete-marker"

I've tried to remove replication and create it again. This action did not help.

Current Behavior

Currently I see this file replication statues:

# mc stat local/test/file-2.yaml
Name      : zbx_export_templates-2.yaml
Date      : 2021-01-23 09:20:01 CET
Size      : 15 KiB
ETag      : 6c8ced0e9f644b94542defeff66372c3
VersionID : 9044152a-7c52-4142-be07-fe9600f4b274
Type      : file
Metadata  :
  Content-Type: application/x-yaml
Replication Status: FAILED


# mc stat local/test/file-36.xml
Name      : zbx_export_templates-36.xml
Date      : 2021-01-24 16:50:01 CET
Size      : 34 KiB
ETag      : 333ca4bb0e6a53bb2d3337765a0e03cb
VersionID : 4183c14d-5448-4aaa-b0c0-cb1878b25ca1
Type      : file
Metadata  :
  Content-Type: text/xml
Replication Status: PENDING

Replication parameters:

# mc replicate export local/test
{
 "Rules": [
  {
   "ID": "c06pctnckkkjebv3ihog",
   "Status": "Enabled",
   "Priority": 1,
   "DeleteMarkerReplication": {
    "Status": "Enabled"
   },
   "DeleteReplication": {
    "Status": "Enabled"
   },
   "Destination": {
    "Bucket": "arn:aws:s3:::test",
    "StorageClass": "STANDARD"
   },
   "Filter": {
    "And": {},
    "Tag": {}
   },
   "SourceSelectionCriteria": {
    "ReplicaModifications": {
     "Status": ""
    }
   }
  }
 ],
 "Role": "arn:minio:replication:ams01:ec22934bb052ca100db62c883efba31df89547bd2705e521dded84fbccac40bb:test"
}

Regression

Yes

My Environment

minio: RELEASE.2021-01-16T02-19-44Z inside docker container

OS: CentOS Linux release 8.2.2004 Kernel: 4.18.0-193.14.2.el8_2.x86_64

closed time in 3 hours

pavel-z1

Pull request review commentminio/minio

Fix replication regression

 func isProxyable(ctx context.Context, bucket string) bool { 	dest := cfg.GetDestination() 	return dest.Bucket == bucket }+ func proxyHeadToRepTarget(ctx context.Context, bucket, object string, opts ObjectOptions) (tgt *TargetClient, oi ObjectInfo, proxy bool, err error) { 	// this option is set when active-active replication is in place between site A -> B, 	// and site B does not have the object yet.-	if opts.ProxyRequest { // true only when site B sets MinIOSourceProxyRequest header+	if opts.ProxyRequest || (opts.ProxyHeaderSet && !opts.ProxyRequest) { // true only when site B sets MinIOSourceProxyRequest header

purpose of proxying when 2 way replication is set up is for automatic failover if one cluster has the object but not yet replicated to other. Not sure what role storage class plays here?

okay, I understood now the relation between replication & proxying

poornas

comment created time in 3 hours

issue commentminio/minio

minio gateway stuck with tls closed connection

will do, but

  • it happened after six weeks, so current version will always be outdated next time
  • went through all release notes since and nothing was mentioned in that regard.

so i think it is worth at least a quick glance

onip

comment created time in 4 hours

issue commentminio/minio

mirror with --watch parameter: "Unable to list comparison retrying.. XML syntax error on line 2: expected attribute name in element"

Do you have a proxy here between mc and MinIO @4001982248998 ?

4001982248998

comment created time in 4 hours

issue commentminio/minio

Minio as Nexus blob store gives Idle timeout expires for larger files

Are you using erasure coded backend or single disk?

rjshree

comment created time in 4 hours

issue closedminio/minio

minio gateway stuck with tls closed connection

<!--- Provide a general summary of the issue in the Title above -->

we're using MinIO as an S3 gateway, through docker, at version RELEASE.2020-06-22T03-12-50Z. today, with an uptime of ~6weeks it stopped responding to our data upload service with this error

API: PutObject(bucket=<omitted_bucket_name>, object=<omitted_object_name>.png)
Time: 07:52:46 UTC 01/24/2021
DeploymentID: e4746081-ba8e-463e-be4a-992db341094d
RequestID: 165D1BC2F21C1131
RemoteHost: 192.168.112.35
Host: minio:9000
UserAgent: MinIO (linux; x64) minio-js/7.0.16
Error: Put https://<omitted_bucket_name>.s3.dualstack.ap-east-1.amazonaws.com/<omitted_object_name>.png: tls: use of closed connection
       4: github.com/minio/minio@/cmd/api-errors.go:1875:cmd.toAPIErrorCode()
       3: github.com/minio/minio@/cmd/api-errors.go:1900:cmd.toAPIError()
       2: github.com/minio/minio@/cmd/object-handlers.go:1486:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()

uname -a

Linux mtr 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I've searched the issues and found nothing similar, hope it is not a duplicate.

closed time in 4 hours

onip

issue commentminio/minio

minio gateway stuck with tls closed connection

Please upgrade to latest @onip and let us know how it goes.

onip

comment created time in 4 hours

push eventminio/minio

Harshavardhana

commit sha e019f21bdaec7f1e2649a324c32c031a59a67b5b

fix: trigger heal if one of the parts are not found (#11358) Previously we added heal trigger when bit-rot checks failed, now extend that to support heal when parts are not found either. This healing gets only triggered if we can successfully decode the object i.e read quorum is still satisfied for the object.

view details

push time in 4 hours

PR merged minio/minio

Reviewers
fix: trigger heal if one of the parts are not found

Description

fix: trigger heal if one of the parts are not found

Motivation and Context

Previously we added heal trigger when bit-rot checks failed, now extend that to support heal when parts are not found either. This healing gets only triggered if we can successfully decode the object i.e read quorum is still satisfied for the object.

How to test this PR?

Just like bit-rot heal, remove some parts and check the object gets healed properly.

Types of changes

  • [x] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Optimization (provides speedup with no functional changes)
  • [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • [ ] Fixes a regression (If yes, please add commit-id or PR # here)
  • [ ] Documentation updated
  • [ ] Unit tests added/updated
+86 -64

1 comment

8 changed files

harshavardhana

pr closed time in 4 hours

Pull request review commentminio/minio

Fix replication regression

 func proxyHeadToRepTarget(ctx context.Context, bucket, object string, opts Objec 	} 	tgt = globalBucketTargetSys.GetRemoteTargetClient(ctx, cfg.RoleArn) 	if tgt == nil || tgt.isOffline() {-		return nil, oi, false, fmt.Errorf("missing target")+		return nil, oi, false, fmt.Errorf("target is offline or not configured")+	}+	if !tgt.replicateSync {

Actually, this check is not required - removing it.

poornas

comment created time in 4 hours

issue commentminio/minio

GCS Gateway: PutObject: Error: storage: object doesn't exist

@dperezro

The issue was originally filed for GCS anyways.

I'm seeing this issue in google as well.

Did you mean some other gw as well, or did you mean you still see it on GCS gw?

Could you give us more info and the error msg you see? What is your setup? Do you have a way to reproduce it?

shamil

comment created time in 4 hours

pull request commentminio/minio

xl: Create a delete-marker when no other version exists

Mint Automation

Test Result
mint-fs.sh :heavy_check_mark:
mint-gateway-s3.sh :heavy_check_mark:
mint-erasure.sh :heavy_check_mark:
mint-dist-erasure.sh :heavy_check_mark:
mint-gateway-nas.sh :heavy_check_mark:
mint-large-bucket.sh :x: more...
mint-gateway-azure.sh :x: more...
mint-zoned.sh :x: more...

<a name="f6992fa8-7577-4df5-8895-f602c1a1090d"></a>

11362-4c16310/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:31951
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp df7c1bb0dca4:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 11 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 55 seconds
(8/15) Running minio-dotnet tests ... done in 46 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 11 seconds
(10/15) Running minio-java tests ... FAILED in 1 minutes and 14 seconds
{
  "name": "minio-java",
  "function": "getObject()",
  "args": "[multi-part upload, offset, length]",
  "duration": 353,
  "status": "FAIL",
  "error": "java.net.SocketException: Connection reset >>> [java.base/java.net.SocketInputStream.read(SocketInputStream.java:186), java.base/java.net.SocketInputStream.read(SocketInputStream.java:140), okio.InputStreamSource.read(JvmOkio.kt:90), okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129), okio.RealBufferedSource.indexOf(RealBufferedSource.kt:449), okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:333), okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29), okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178), okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106), okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201), okhttp3.internal.connection.RealCall.execute(RealCall.kt:154), io.minio.MinioClient.execute(MinioClient.java:647), io.minio.MinioClient.uploadPart(MinioClient.java:4747), io.minio.MinioClient.putObject(MinioClient.java:2738), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testGetObject(FunctionalTest.java:1136), FunctionalTest.getObject(FunctionalTest.java:1229), FunctionalTest.runObjectTests(FunctionalTest.java:3752), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... done in 50 seconds
(11/15) Running minio-py tests ... done in 3 minutes and 16 seconds
(12/15) Running s3cmd tests ... done in 22 seconds
(13/15) Running s3select tests ... done in 6 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

<a name="f5ee9266-e61f-46bb-b13d-6c3ea3dc531c"></a>

11362-4c16310/mint-zoned.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:31834
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 2928e56919a2:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 1 minutes and 24 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 32 seconds
(8/15) Running minio-dotnet tests ... done in 56 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 33 seconds
{
  "args": {
    "bucketName": "minio-go-test-wtjyga39rrjqki6u",
    "objectName": "ce1ci5ni6td3sgfev1rgj1aiw1amoc",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 5269,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjects(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjects",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 1 minutes and 54 seconds
(10/15) Running minio-js tests ... done in 55 seconds
(11/15) Running minio-py tests ... done in 3 minutes and 44 seconds
(12/15) Running s3cmd tests ... done in 22 seconds
(13/15) Running s3select tests ... done in 8 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

<a name="a132642d-09e5-45a5-ba99-9de3f9b941d6"></a>

11362-4c16310/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:30207
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp d9252658a986:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 11 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 4 minutes and 16 seconds
(4/15) Running aws-sdk-ruby tests ... done in 21 seconds
(5/15) Running awscli tests ... done in 2 minutes and 55 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 16 minutes and 48 seconds
(8/15) Running minio-dotnet tests ... done in 3 minutes and 7 seconds
(9/15) Running minio-go tests ... done in 20 minutes and 55 seconds
(10/15) Running minio-java tests ... FAILED in 9 minutes and 4 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 171,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:2ff85b27-d01e-0058-34cb-f4e995000000\nTime:2021-01-27T16:43:28.0143387Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'Bt2TMNrem9UW6Tga+H1pVQ7fjhKZla5NBwtfLk3YcOk=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:c51c71f4-348e-41c6-693d-1b6b58a07d1d\nx-ms-date:Wed, 27 Jan 2021 16:43:27 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商品\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-33i2kl3/minio-java-test-36gv99k\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-33i2kl3/minio-java-test-36gv99k?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2021-01-27T15:39:32Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [c51c71f4-348e-41c6-693d-1b6b58a07d1d]\n   X-Ms-Date: [Wed, 27 Jan 2021 16:43:27 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Wed, 27 Jan 2021 16:43:27 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [2ff85b27-d01e-0058-34cb-f4e995000000]\n\n\n, bucketName = minio-java-test-33i2kl3, objectName = minio-java-test-36gv99k, resource = /minio-java-test-33i2kl3/minio-java-test-36gv99k, requestId = 165E24767781B029, hostId = 8a43707d-ff87-4e84-9dfc-7e2553429a6e)\nrequest={method=PUT, url=http://minio-dev6.minio.io:30207/minio-java-test-33i2kl3/minio-java-test-36gv99k, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-dev6.minio.io:30207\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20210127T164327Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210127/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 165E24767781B029\nX-Xss-Protection: 1; mode=block\nDate: Wed, 27 Jan 2021 16:43:28 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... FAILED in 50 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 79,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:79:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (internal/streams/readable.js:684:10) at endReadableNT (internal/streams/readable.js:1327:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(10/15) Running minio-py tests ... done in 25 minutes and 32 seconds
(11/15) Running s3cmd tests ... done in 10 minutes and 6 seconds
(12/15) Running s3select tests ... done in 1 minutes and 4 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub Deleting image locally

vadmeste

comment created time in 5 hours

Pull request review commentminio/minio

Fix replication regression

 func isProxyable(ctx context.Context, bucket string) bool { 	dest := cfg.GetDestination() 	return dest.Bucket == bucket }+ func proxyHeadToRepTarget(ctx context.Context, bucket, object string, opts ObjectOptions) (tgt *TargetClient, oi ObjectInfo, proxy bool, err error) { 	// this option is set when active-active replication is in place between site A -> B, 	// and site B does not have the object yet.-	if opts.ProxyRequest { // true only when site B sets MinIOSourceProxyRequest header+	if opts.ProxyRequest || (opts.ProxyHeaderSet && !opts.ProxyRequest) { // true only when site B sets MinIOSourceProxyRequest header

purpose of proxying when 2 way replication is set up is for automatic failover if one cluster has the object but not yet replicated to other. Not sure what role storage class plays here?

poornas

comment created time in 5 hours

pull request commentminio/minio

Heal reporting

Mint Automation

Test Result
mint-large-bucket.sh :heavy_check_mark:
mint-fs.sh :heavy_check_mark:
mint-gateway-s3.sh :heavy_check_mark:
mint-erasure.sh :heavy_check_mark:
mint-dist-erasure.sh :heavy_check_mark:
mint-zoned.sh :heavy_check_mark:
mint-gateway-nas.sh :heavy_check_mark:
mint-gateway-azure.sh :x: more...

<a name="d202a325-83eb-4ab7-a8e7-9b9b57580a45"></a>

11312-13303b7/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:32477
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 1ad47a7c16b9:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 10 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 4 minutes and 19 seconds
(4/15) Running aws-sdk-ruby tests ... done in 27 seconds
(5/15) Running awscli tests ... done in 2 minutes and 53 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 16 minutes and 43 seconds
(8/15) Running minio-dotnet tests ... done in 4 minutes and 10 seconds
(9/15) Running minio-go tests ... FAILED in 12 minutes and 0 seconds
{
  "args": {
    "bucketName": "minio-go-test-ujd0t3mewa0cmeie",
    "objectName": "316kabopuabwxorexitx4ywt2f69if"
  },
  "duration": 1006,
  "error": "EOF",
  "function": "GetObject(bucketName, objectName)",
  "message": "Error reading through readFull",
  "name": "minio-go: testGetObjectSeekEnd",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 9 minutes and 0 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 171,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:18fa3541-101e-0168-52ba-f4f5ea000000\nTime:2021-01-27T14:38:47.8518793Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 's5stq6jlBAyWB7qDh7TQQwfqD/3M9HDk1kYJ35Opa7Q=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:136d04a0-dbba-44f4-7ea2-292895a648ad\nx-ms-date:Wed, 27 Jan 2021 14:38:47 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商品\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-oa7bnt/minio-java-test-327m9tk\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-oa7bnt/minio-java-test-327m9tk?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2021-01-27T13:42:43Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [136d04a0-dbba-44f4-7ea2-292895a648ad]\n   X-Ms-Date: [Wed, 27 Jan 2021 14:38:47 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1091]\n   Content-Type: [application/xml]\n   Date: [Wed, 27 Jan 2021 14:38:46 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [18fa3541-101e-0168-52ba-f4f5ea000000]\n\n\n, bucketName = minio-java-test-oa7bnt, objectName = minio-java-test-327m9tk, resource = /minio-java-test-oa7bnt/minio-java-test-327m9tk, requestId = 165E1DA8DD610BFF, hostId = 1b06aea7-e2fa-42ee-bcef-8653ffc3d3c1)\nrequest={method=PUT, url=http://minio-c2.minio.io:32477/minio-java-test-oa7bnt/minio-java-test-327m9tk, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-c2.minio.io:32477\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20210127T143847Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210127/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3082\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 165E1DA8DD610BFF\nX-Xss-Protection: 1; mode=block\nDate: Wed, 27 Jan 2021 14:38:47 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(9/15) Running minio-js tests ... done in 24 minutes and 38 seconds
(10/15) Running minio-py tests ... done in 23 minutes and 8 seconds
(11/15) Running s3cmd tests ... done in 10 minutes and 34 seconds
(12/15) Running s3select tests ... done in 1 minutes and 5 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub Deleting image locally

klauspost

comment created time in 7 hours

issue commentminio/minio

minio gateway stuck with tls closed connection

forgot to mention that restarting the container

$ docker restart minio

fixed the problem.

onip

comment created time in 7 hours

issue openedminio/minio

minio gateway stuck with tls closed connection

<!--- Provide a general summary of the issue in the Title above -->

we're using MinIO as an S3 gateway, through docker, at version RELEASE.2020-06-22T03-12-50Z. today, with an uptime of ~6weeks it stopped responding to our data upload service with this error

API: PutObject(bucket=<omitted_bucket_name>, object=<omitted_object_name>.png)
Time: 07:52:46 UTC 01/24/2021
DeploymentID: e4746081-ba8e-463e-be4a-992db341094d
RequestID: 165D1BC2F21C1131
RemoteHost: 192.168.112.35
Host: minio:9000
UserAgent: MinIO (linux; x64) minio-js/7.0.16
Error: Put https://<omitted_bucket_name>.s3.dualstack.ap-east-1.amazonaws.com/<omitted_object_name>.png: tls: use of closed connection
       4: github.com/minio/minio@/cmd/api-errors.go:1875:cmd.toAPIErrorCode()
       3: github.com/minio/minio@/cmd/api-errors.go:1900:cmd.toAPIError()
       2: github.com/minio/minio@/cmd/object-handlers.go:1486:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()

uname -a

Linux mtr 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I've searched the issues and found nothing similar, hope it is not a duplicate.

created time in 7 hours

issue commentminio/minio

Cluster nodes are hanging / not responding to requests

We have a total of 11 distributed deployments in different locations around the world that we updated to the latest version and we've seen this issue in three of them so far, so something environment specific like packet loss seems unlikely to be happening in all of them.

I'll run the commands the next time we see errors and upload the files.

amhuber

comment created time in 7 hours

more