公司网站建设设计公司,wordpress主题使用加,微网站免费制作,wordpress search页面缘起
今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求#xff0c;由于4月份自学考试临近#xff0c;一直在备考就拖着没总结记录下#xff0c;开发联调过程中也出现过一些奇葩的问题#xff0c;最近人刚从考试缓过来顺手记录一下。
S3对象存储的基本概念
…缘起
今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求由于4月份自学考试临近一直在备考就拖着没总结记录下开发联调过程中也出现过一些奇葩的问题最近人刚从考试缓过来顺手记录一下。
S3对象存储的基本概念
S3是什么
Amazon S3(Simple Storage Service)对象存储出现得比较早且使用简单的RESTful API于是成为了对象存储服务(Object Storage ServiceOSS)业内的标准接口规范。
S3的逻辑模型
如下图我们可以把S3的存储空间想象成无限的想存储一个任意格式的文件到S3服务中只需要知道要把它放到哪个桶(Bucket)中它的名字Object Id应该是什么。 按图中的模型可简单理解为S3是由若干个桶Bucket组成每个桶中包含若干个不同标识的对象Object还有就是统一的访问入口(RESTful API)这样基本就足够了。
Minio客户端方式操作S3
详细API文档https://min.io/docs/minio/linux/developers/java/API.html
以下代码异常处理做了简化真实使用时请注意捕获异常做处理。
引入依赖
Maven:
dependencygroupIdio.minio/groupIdartifactIdminio/artifactIdversion8.5.2/version
/dependencyGradle:
dependencies {implementation(io.minio:minio:8.5.2)
}初始化客户端
private static final String HTTP_PROTOCOL http;private MinioClient minioClient;
private String endpoint http://192.168.0.8:9200;
private String accessKey testKey;
private String secretKey testSecretKey;public void init() throws MalformedURLException {URL endpointUrl new URL(endpoint);try {// url上无端口号时识别http为80端口https为443端口int port endpointUrl.getPort() ! -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();boolean security HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;//formatter:offthis.minioClient MinioClient.builder().endpoint(endpointUrl.getHost(), port, security).credentials(accessKey, secretKey).build();//formatter:on// 忽略证书校验防止自签名证书校验失败导致无法建立连接this.minioClient.ignoreCertCheck();} catch (Exception e) {e.printStackTrace();}
}建桶
public boolean createBucket(String bucket) {try {minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();return false;}return true;
}删桶
public boolean deleteBucket(String bucket) {try {minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());logger.info(删除桶[{}]成功, bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;
}判断桶是否存在
public boolean bucketExists(String bucket) {try {return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();return false;}
}上传对象
public void upload(String bucket, String objectId, InputStream input) {try {//formatter:offminioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId).stream(input, input.available(), -1).build());//formatter:on} catch (Exception e) {e.printStackTrace();}
}下载对象
提供两个下载方法一个将输入流返回另一个用参数输出流写出
public InputStream download(String bucket, String objectId) {try {return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}return null;
}public void download(String bucket, String objectId, OutputStream output) {//formatter:offtry (InputStream input minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {IOUtils.copyLarge(input, output);} catch (Exception e) {e.printStackTrace();}//formatter:on
}删除对象
public boolean deleteObject(String bucket, String objectId) {//formatter:offtry {minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}//formatter:onreturn true;
}判断对象是否存在
public boolean objectExists(String bucket, String key) {//formatter:offtry {// minio客户端未提供判断对象是否存在的方法此方法中调用出现异常时说明对象不存在minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(key).build());} catch (Exception e) {return false;}//formatter:onreturn true;
}完整代码
import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import io.minio.BucketExistsArgs;
import io.minio.GetObjectArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.RemoveBucketArgs;
import io.minio.RemoveObjectArgs;
import io.minio.StatObjectArgs;public class S3MinioClientDemo {private static final Logger logger LoggerFactory.getLogger(S3MinioClientDemo.class);private static final String HTTP_PROTOCOL http;private MinioClient minioClient;private String endpoint http://192.168.0.8:9200;private String accessKey testKey;private String secretKey testSecretKey;public void init() throws MalformedURLException {URL endpointUrl new URL(endpoint);try {// url上无端口号时识别http为80端口https为443端口int port endpointUrl.getPort() ! -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();boolean security HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;//formatter:offthis.minioClient MinioClient.builder().endpoint(endpointUrl.getHost(), port, security).credentials(accessKey, secretKey).build();//formatter:on// 忽略证书校验防止自签名证书校验失败导致无法建立连接this.minioClient.ignoreCertCheck();} catch (Exception e) {e.printStackTrace();}}public boolean createBucket(String bucket) {try {boolean found minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());if (found) {logger.info(桶名[{}]已存在, bucket);return false;}minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();}return true;}public boolean deleteBucket(String bucket) {try {minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());logger.info(删除桶[{}]成功, bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;}public boolean bucketExists(String bucket) {try {return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();return false;}}public void upload(String bucket, String objectId, InputStream input) {try {//formatter:offminioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId).stream(input, input.available(), -1).build());//formatter:on} catch (Exception e) {e.printStackTrace();}}public InputStream download(String bucket, String objectId) {try {return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}return null;}public void download(String bucket, String objectId, OutputStream output) {//formatter:offtry (InputStream input minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {IOUtils.copyLarge(input, output);} catch (Exception e) {e.printStackTrace();}//formatter:on}public boolean objectExists(String bucket, String objectId) {//formatter:offtry {// minio客户端未提供判断对象是否存在的方法此方法中调用出现异常时说明对象不存在minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {return false;}//formatter:onreturn true;}public boolean deleteObject(String bucket, String objectId) {//formatter:offtry {minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}//formatter:onreturn true;}public void close() {minioClient null;}}Amazon S3 SDK方式操作S3
官方API文档https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
这里由于项目上提供的SDK和文档都是1.x的这里就暂时只提供1.x的代码
引入依赖
Maven:
dependencygroupIdcom.amazonaws/groupIdartifactIdaws-java-sdk-s3/artifactIdversion1.11.300/version
/dependencyGradle:
dependencies { implementation com.amazonaws:aws-java-sdk-s3:1.11.300
}初始化客户端
private static final Logger logger LoggerFactory.getLogger(S3SdkDemo.class);private AmazonS3 s3client;
private String endpoint http://192.168.0.8:9200;
private String accessKey testKey;
private String secretKey testSecretKey;public void init() throws MalformedURLException {URL endpointUrl new URL(endpoint);String protocol endpointUrl.getProtocol();int port endpointUrl.getPort() -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();ClientConfiguration clientConfig new ClientConfiguration();clientConfig.setSignerOverride(S3SignerType);clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));// 禁用证书检查避免https自签证书校验失败System.setProperty(com.amazonaws.sdk.disableCertChecking, true);// 屏蔽 AWS 的 MD5 校验避免校验导致的下载抛出异常问题System.setProperty(com.amazonaws.services.s3.disableGetObjectMD5Validation, true);AWSCredentials awsCredentials new BasicAWSCredentials(accessKey, secretKey);// 创建 S3Client 实例AmazonS3 s3client new AmazonS3Client(awsCredentials, clientConfig);s3client.setEndpoint(endpointUrl.getHost() : port);s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());this.s3client s3client;
}建桶
public boolean createBucket(String bucket) {String bucketName parseBucketName(bucket);try {if (s3client.doesBucketExist(bucketName)) {logger.warn(bucket[{}]已存在, bucketName);return false;}s3client.createBucket(bucketName);} catch (Exception e) {e.printStackTrace();}return true;
}删桶
public boolean deleteBucket(String bucket) {try {s3client.deleteBucket(bucket);logger.info(删除bucket[{}]成功, bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;
}判断桶是否存在
public boolean bucketExists(String bucket) {try {return s3client.doesBucketExist(bucket);} catch (Exception e) {e.printStackTrace();}return false;
}上传对象
public void upload(String bucket, String objectId, InputStream input) {try {// 创建文件上传的元数据ObjectMetadata meta new ObjectMetadata();// 设置文件上传长度meta.setContentLength(input.available());// 上传s3client.putObject(bucket, objectId, input, meta);} catch (Exception e) {e.printStackTrace();}
}下载对象
public InputStream download(String bucket, String objectId) {try {S3Object o s3client.getObject(bucket, objectId);return o.getObjectContent();} catch (Exception e) {e.printStackTrace();}return null;
}public void download(String bucket, String objectId, OutputStream out) {S3Object o s3client.getObject(bucket, objectId);try (InputStream in o.getObjectContent()) {IOUtils.copyLarge(in, out);} catch (Exception e) {e.printStackTrace();}
}删除对象
public boolean deleteObject(String bucket, String objectId) {try {s3client.deleteObject(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}return true;
}判断对象是否存在
public boolean existObject(String bucket, String objectId) {try {return s3client.doesObjectExist(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}
}完整代码
import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.S3ClientOptions;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3Object;/*** S3对象存储官方SDK实现** author ZhangChenguang* date 2023年2月2日*/
SuppressWarnings(deprecation)
public class S3SdkDemo {private static final Logger logger LoggerFactory.getLogger(S3SdkDemo.class);private AmazonS3 s3client;private String endpoint http://192.168.0.8:9200;private String accessKey testKey;private String secretKey testSecretKey;public void init() throws MalformedURLException {URL endpointUrl new URL(endpoint);String protocol endpointUrl.getProtocol();int port endpointUrl.getPort() -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();ClientConfiguration clientConfig new ClientConfiguration();clientConfig.setSignerOverride(S3SignerType);clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));// 禁用证书检查避免https自签证书校验失败System.setProperty(com.amazonaws.sdk.disableCertChecking, true);// 屏蔽 AWS 的 MD5 校验避免校验导致的下载抛出异常问题System.setProperty(com.amazonaws.services.s3.disableGetObjectMD5Validation, true);AWSCredentials awsCredentials new BasicAWSCredentials(accessKey, secretKey);// 创建 S3Client 实例AmazonS3 s3client new AmazonS3Client(awsCredentials, clientConfig);s3client.setEndpoint(endpointUrl.getHost() : port);s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());this.s3client s3client;}public boolean createBucket(String bucket) {try {s3client.createBucket(bucket);} catch (Exception e) {e.printStackTrace();}return true;}public boolean deleteBucket(String bucket) {try {s3client.deleteBucket(bucket);logger.info(删除bucket[{}]成功, bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;}public boolean bucketExists(String bucket) {try {return s3client.doesBucketExist(bucket);} catch (Exception e) {e.printStackTrace();}return false;}public void upload(String bucket, String objectId, InputStream input) {try {// 创建文件上传的元数据ObjectMetadata meta new ObjectMetadata();// 设置文件上传长度meta.setContentLength(input.available());// 上传s3client.putObject(bucket, objectId, input, meta);} catch (Exception e) {e.printStackTrace();}}public InputStream download(String bucket, String objectId) {try {S3Object o s3client.getObject(bucket, objectId);return o.getObjectContent();} catch (Exception e) {e.printStackTrace();}return null;}public void download(String bucket, String objectId, OutputStream out) {S3Object o s3client.getObject(bucket, objectId);try (InputStream in o.getObjectContent()) {IOUtils.copyLarge(in, out);} catch (Exception e) {e.printStackTrace();}}public boolean existObject(String bucket, String objectId) {try {return s3client.doesObjectExist(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}}public boolean deleteObject(String bucket, String objectId) {try {s3client.deleteObject(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}return true;}public void close() {s3client null;}
}遇到的问题
1、bucket名称必须是小写不支持下划线
处理方式写方法转换下bucket名称将大写转小写将下划线替换为中划线。
2、minio客户端下载非官方S3存储的文件时如果响应头的Content-Length与实际文件大小不符会导致minio客户端包装的okhttp3报错
报错信息
Caused by: java.net.ProtocolException: unexpected end of streamat okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430) ~[okhttp-3.14.9.jar:?]at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[okhttp-3.14.9.jar:?]at okio.RealBufferedSource$1.read(RealBufferedSource.java:447) ~[okio-1.17.2.jar:?]at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83) ~[nr.file-2.5.7.jar:?]at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395) ~[nr.file-2.5.7.jar:?]... 122 more抓包发现问题的图 最终换成了S3官方SDK可用了。
PS客户现场部署的S3是浪潮公司提供的如果现场遇到这个情况就不要固执去找对方对线了完全没用。。
总结
S3存储的基本操作就记录到这里了由于没有S3存储就没尝试官方SDK的V2版本由于这些代码是总结时从业务代码里抽取出来的可能会有点问题但大体思路已经有了。
希望对读者有所用处觉得写得不错和有帮到你欢迎点个赞您的支持就是我的鼓励
